What is a checkpoint?
A checkpoint is a copy of a model’s weights written to disk partway through training, typically every few hundred steps or at the end of each epoch. A full checkpoint also includes the optimizer’s state and the position in the data, so training can resume exactly where it stopped. A weights-only checkpoint is smaller and is what gets kept for evaluation and deployment.
Model publishers also call each released version of a model a checkpoint, and a fine-tuned variant is described as a checkpoint derived from a base model.
Why checkpoints matter
Checkpoints protect against three things. The first is failure. A run that takes days on rented graphics processing units (GPUs) will eventually hit a hardware fault, and a recent checkpoint turns a lost week into a lost hour. The second is overfitting. When validation loss starts rising after epoch four, the checkpoint from epoch three is the model you want, and it exists only if it was saved. The third is accountability. A checkpoint with its training configuration and evaluation results recorded in a model registry is the artifact an auditor can point to.
Checkpoint in practice
A logistics company fine-tunes a model that reads bills of lading. Training runs overnight on a cloud GPU and the instance is reclaimed at two in the morning. The engineer restarts from the checkpoint saved twenty minutes earlier and the run finishes by breakfast. Each checkpoint is scored on a held-out set, and the best one, rather than the last one, is promoted.