What is model robustness?
Model robustness is the degree to which a model keeps performing correctly when its inputs differ from the clean examples it was trained and tested on. Production data is messier than test data. Scans are skewed, sensor readings are noisy, customers misspell product names, and required fields come in empty. Over time the population itself shifts through data drift. Some inputs are hostile, crafted by someone who wants a particular wrong answer. A model with high robustness degrades gradually under these pressures and signals low confidence when it is out of its depth, while a fragile model fails abruptly and with confidence.
Robustness extends generalization, which covers ordinary unseen data, to the hardest and most hostile cases. It is measured with stress tests. Teams perturb a test set on purpose with noise, blur, typos, or missing values and track how fast accuracy falls. They measure accuracy on separate slices of data, such as each region or document type, and test against adversarial examples. The result is a profile of where the model holds up and where it breaks.
Model robustness in practice
A freight brokerage extracts weights, addresses, and reference numbers from bills of lading. Its model scored well on clean PDFs from large carriers. In production, many documents arrive as phone photos taken in truck cabs, at an angle, in poor light, with creases and handwriting. The team builds a test set of degraded images, measures field accuracy under each condition, adds rotated and darkened images to the training data, and sends low-confidence extractions to a person for review.
Model robustness in the EU AI Act
Article 15 of the EU AI Act requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity and to perform consistently throughout their lifecycle. It names specific threats to address where appropriate, including data poisoning and adversarial examples. Providers of high-risk systems must therefore record their robustness testing and metrics in the system’s technical documentation.
What to watch for
- Ask vendors for accuracy on degraded and unusual inputs, because a clean test set says little about production.
- Review the worst-performing slice of data alongside the average, since that slice is where failures and complaints come from.
- Expect a trade-off, because training a model to resist attacks can lower its accuracy on ordinary inputs.
- Repeat robustness tests after each retraining, since a new version can be fragile where the old one held up.