What is direct preference optimization?
Direct preference optimization (DPO) is a training technique that teaches a language model which of two responses people prefer, using the preference pairs directly as training data. It pursues the same goal as reinforcement learning from human feedback (RLHF), which works in two stages: train a reward model on human comparisons, then run reinforcement learning against it. DPO collapses the two into one. Given a prompt, a preferred response, and a rejected response, it adjusts the weights so the preferred response becomes more likely and the rejected one less likely, relative to a frozen copy of the original model that keeps it from drifting.
The procedure looks like ordinary supervised fine-tuning, needs no reward model or sampling loop, and runs on the same hardware a company already uses for fine-tuning with LoRA adapters.
Why direct preference optimization matters
DPO makes preference tuning practical for a company running its own open-weight model. A law firm with an assistant that drafts client correspondence can have partners mark which of two drafts they prefer. A few thousand such pairs teach the model the firm’s tone, structure, and level of caution without anyone writing rules for them.
What to watch for
- Preference data quality decides the outcome; inconsistent raters produce a model that is confidently inconsistent.
- Narrow preference data can degrade general ability, so evaluate on a broad test set afterward against the baseline model.
- Pairs collected from production conversations may contain customer data and need the same handling as any other training data.