What is knowledge distillation?
Knowledge distillation is a technique for training a smaller student model to imitate a larger teacher model, so that the student runs faster and on cheaper hardware while keeping most of the teacher’s accuracy. Instead of learning only from the original labels, the student learns from the teacher’s outputs, which carry more information. A teacher that says an image is probably a cat but somewhat dog-like teaches the student about similarity between classes that a plain label cannot convey.
How knowledge distillation works
For classifiers, the teacher produces probability distributions over the classes for every training example, and the student is trained to match those distributions as well as the true labels. For language models, distillation usually means generating data. The teacher answers a large set of prompts, often with its reasoning written out, and the student is fine-tuned on those answers. Many small open-weight language models were trained this way from larger ones. Distillation differs from quantization, which shrinks a model by storing its weights with less precision, and from pruning, which removes weights. The three are often combined.
Knowledge distillation in practice
A legal services firm receives tens of thousands of documents a month that must be sorted by matter type and urgency. A large language model does this well, but running it on every page is slow and expensive. The firm uses the large model to label its historical documents, reviews a sample of the labels, and fine-tunes a small model on the result. The small model runs on a single GPU in the firm’s own data center, handles the routine volume in milliseconds, and routes the ambiguous cases to the large model. Kirk Tech Solutions typically uses this teacher-and-student pattern when a client has one high-volume task that does not need a general-purpose model.
What to watch for
- Check the teacher’s license, because many model licenses restrict using outputs to train other models.
- The student inherits the teacher’s mistakes and biases, so review a sample of the teacher’s labels before training.
- Evaluate the student on your own data rather than on public benchmarks, since distillation narrows what the model can do.
- Keep a path to the teacher for hard cases, typically through model routing by confidence.