What is LLM-as-a-judge?
LLM-as-a-judge is the use of a large language model to evaluate the output of an AI system, assigning a score, a pass or fail, or a preference between two candidates according to written criteria. The judge model receives the original input, the response under test, often a reference answer, and a rubric that explains what good looks like. It returns a verdict and usually a short justification. Because the judge is itself a model, it can evaluate thousands of responses in minutes at low cost, where human reviewers would take weeks.
The approach exists because many generative tasks have no single correct answer. A summary can be accurate and complete in different words than the reference. Exact-match metrics fail on this, and human review does not scale. A judge model is a middle path that applies judgment, imperfectly, at machine speed.
How LLM-as-a-judge works
A team writes the rubric first, with specific criteria such as “every figure in the summary appears in the source” or “the reply includes the required disclosure.” They collect a sample of a few hundred responses and have people grade them. The judge model grades the same sample, and the team compares the two until the judge agrees with the humans often enough to be trusted, adjusting the prompt and criteria as needed. From then on the judge runs on every new model version, prompt change, or production sample, with a small human audit to make sure agreement has not drifted.
LLM-as-a-judge in practice
A law firm uses a model to produce first-draft summaries of deposition transcripts. Before each change to the prompt or model, the team runs the new version against two hundred transcripts and has a judge model check each summary for invented facts, missing parties, and wrong dates, with the transcript as reference. Associates still review the real summaries, but the judge tells the team whether a change made things better or worse before anyone’s time is spent.
What to watch for
- Judges share the biases of the models they evaluate; they favor longer answers, confident tone, and responses from models similar to themselves.
- Calibrate the judge against human grades on your own data before trusting its scores, and re-check the agreement periodically.
- Use a different model as the judge than the one being tested where possible, and ask for a justification so a person can spot-check the reasoning.
- A judge is an evaluation tool, so avoid using its score alone to approve an output for a customer without human oversight on high-stakes tasks.