What is LoRA (Low-Rank Adaptation)?
LoRA (Low-Rank Adaptation) is a technique for fine-tuning a large model without changing its original weights. Instead of updating the billions of numbers in the base model, LoRA freezes them and inserts a pair of small matrices alongside selected layers, usually the attention layers. Only those small matrices are trained. The term low-rank describes their shape: they are thin, so they hold a tiny fraction of the parameters of the layer they modify, yet their product can shift the layer’s output enough to teach new behavior.
The trained matrices form an adapter, typically tens to a few hundred megabytes, compared with tens of gigabytes for the full model. At inference time the adapter is either merged into the base weights or loaded alongside them. A variant called QLoRA trains the adapter on top of a quantized base model, which lets a single GPU fine-tune a model that would otherwise need several.
Why LoRA matters
Full fine-tuning of a large model needs a cluster, days of time, and a complete copy of the model for every variant. LoRA brings fine-tuning within reach of a company with one or two GPUs and a few thousand examples. Because the base model is untouched, one deployment can serve many adapters, swapping them per request. It also reduces the risk of catastrophic forgetting, where a fully fine-tuned model loses general ability while learning a narrow task. LoRA is the most widely used form of parameter-efficient fine-tuning (PEFT).
LoRA in practice
A collections agency must produce letters that follow strict regulatory wording and its own house style. Prompting alone gets the format mostly right but drifts on tone and misses required disclosures. The agency trains a LoRA adapter on a few thousand approved letters paired with the account facts that produced them. The adapter runs on the same self-hosted model used for other tasks, and a second adapter trained on call summaries is loaded for the contact center. Compliance reviews the adapter’s outputs against the same checklist used for human drafts.
What to watch for
- LoRA teaches style, format, and domain vocabulary well, but it is a poor way to add facts, so use retrieval for knowledge that changes.
- Training data quality dominates the result, so review the examples as carefully as the output.
- Re-train the adapter when the base model is upgraded, because adapters are tied to the exact weights they were trained on.
- Version adapters and record which one produced each output, since they change behavior as much as a prompt does.