What is reinforcement learning from human feedback?
Reinforcement learning from human feedback (RLHF) is the stage of training that turns a language model from a text predictor into an assistant that answers the way people want. A pre-trained model can continue any passage of text, but it has no sense of which continuation is helpful, honest, or appropriate. RLHF supplies that sense from human judgments. People compare pairs of model responses to the same prompt and pick the better one. Those comparisons train a second model, the reward model, to predict which responses humans will prefer. The language model is then adjusted with reinforcement learning to produce responses the reward model scores highly, while being kept from drifting too far from its original behavior.
How RLHF works
The full pipeline has three steps. First, supervised fine-tuning teaches the model the format of helpful answers from a curated set of example conversations, which is the instruction tuning stage. Second, human raters rank the model’s outputs on thousands of prompts, and the reward model is trained on those rankings. Third, a reinforcement learning algorithm optimizes the model against the reward model, generating responses, scoring them, and nudging the model weights toward higher scores. The process is expensive and delicate, which is why alternatives such as direct preference optimization that skip the separate reward model have become common.
Why RLHF matters
RLHF is the reason current assistants follow instructions, decline harmful requests, and explain themselves in a consistent voice. It is also the source of some of their quirks. A model trained to please raters can become sycophantic, agreeing with a user’s wrong premise or praising weak work, and it may prefer long, hedged answers because raters rewarded them. For a company this matters in two ways. When evaluating models, the behavior you are judging is largely a product of each vendor’s RLHF choices and the raters they used. When fine-tuning your own open-weight model, preference data from your own staff, such as a claims adjuster marking which of two draft letters is better, is how the model learns your standards rather than a vendor’s.
What to watch for
- Thumbs up and down buttons in a chat interface are feedback data; they only become RLHF if someone builds a training pipeline around them, which most companies never do.
- Rater instructions shape the model, so a vendor’s values and policies are embedded in its models’ behavior whether or not they match yours.
- Models tuned to be agreeable need checks against user error, so design workflows in which the model verifies a claim rather than confirming it.