What is model evaluation?
Model evaluation is the process of measuring a model’s performance against criteria that were agreed before the measurement began. For a classic model, it means scoring predictions on test data the model never saw and reporting metrics such as precision and recall for a classifier or mean absolute error for a regression model. For a large language model or an agent there is no single correct answer to score against, so evaluation combines several methods.
Public benchmarks compare models on standard question sets and are useful for a first shortlist. Domain task sets are built from the company’s own work: a few hundred real support tickets with approved responses, or contract clauses with the correct extraction. An automated grader, often an LLM-as-a-judge, scores outputs at scale, and people check a sample to confirm the grader agrees with them. Agents are evaluated end to end on whether they completed the task, how many tool calls it took, and whether any action broke a rule.
Why model evaluation matters
Evaluation is the only evidence that a model does what the project claims. It supports the go-live decision, the choice between an open-weight model and a hosted one, and the proof to an auditor or regulator that the system was tested. Under the EU AI Act, high-risk systems must document their accuracy testing, and model evaluation produces that record.
Safety criteria belong in the same evaluation. A customer-facing assistant is tested for whether it leaks data from other customers, follows instructions planted in a document, or produces harmful content, using red-team prompts kept separate from the capability tests.
Model evaluation in practice
A law firm evaluating a contract-review assistant builds a set of two hundred clauses from closed matters, each with the risk rating a partner assigned. Three candidate models run over the set with the same prompt. The firm measures agreement with the partner ratings, the rate of invented citations, and refusal behavior on privileged content. The smallest model agrees with the partners as often as the largest and is chosen. The same set is re-run after every prompt or model change.
What to watch for
- Build the evaluation set from the company’s own data, because public benchmarks measure general ability rather than the task at hand.
- Keep the evaluation set separate from anything used for prompt tuning or fine-tuning, or the scores stop meaning anything.
- Define pass criteria before running the evaluation so the result cannot be argued into a pass afterward.
- Re-run the full evaluation after every change and treat it as a release gate.