What is pre-training?
Pre-training is the initial phase of training in which a model learns general structure from a very large dataset before it is adapted to any particular job. For a large language model, pre-training means predicting the next token across trillions of tokens of text. For a vision model, it may mean predicting masked patches of an image. No human labels are needed because the data supplies its own answers, which is why pre-training is a form of self-supervised learning.
The result is a base model that is fluent in the patterns of its data but not yet shaped to follow instructions or perform a specific task. Instruction tuning and fine-tuning come afterward and are small by comparison.
Why pre-training matters
Pre-training is where almost all of the compute and cost in modern AI is spent. A frontier language model takes thousands of graphics processing units (GPUs) running for months. No mid-sized company does this, and none needs to. The business decision is which pre-trained model to adopt, which is governed by the model’s license, its size, the languages and domains in its training data, and whether the weights can be downloaded and run privately.
A logistics company choosing between two open-weight models for its dispatch assistant, for example, will care that one was pre-trained on far more multilingual text, because its drivers and shippers write in four languages.