What is a risk management system under the EU AI Act?
A risk management system, in the EU AI Act’s sense, is the documented and continuous process that a provider of a high-risk AI system must establish, run, and keep updated for the system’s whole lifecycle. Article 9 of the Act defines it. The process has to identify the known and reasonably foreseeable risks the system poses to health, safety, and fundamental rights, both when used as intended and under reasonably foreseeable misuse, estimate and evaluate those risks, and adopt measures that bring the remaining risk to an acceptable level.
The phrase is narrower than AI risk management in general. Companies manage AI risk voluntarily for all kinds of systems. The Act’s risk management system is a legal requirement attached to one tier, with content the regulator can inspect.
How a risk management system works
The Act expects the process to be iterative. Risks are identified during design, re-evaluated as testing produces data, and reviewed again using post-market monitoring once the system is in use. Risk treatment follows a hierarchy: remove or reduce the risk through design where possible, add controls where it cannot be removed, and give deployers information and training for what remains. Residual risk must be judged acceptable and the judgment written down.
Testing is part of the system. The provider must test against defined metrics and probabilistic thresholds suited to the intended purpose, before the system reaches the market and throughout development. The Act also requires providers to consider whether the system is likely to affect people under eighteen or other vulnerable groups. For products already covered by sectoral law, such as medical devices, the AI risk process can be folded into the existing risk management procedures.
The risk management system in practice
A vendor sells an AI tool that prioritizes emergency calls for a regional dispatch center, a critical infrastructure use that is high-risk. Its risk file lists the harms: a serious call ranked low, a bias against callers with accents the speech model handles poorly, and dispatchers trusting the ranking without listening. Design measures include a floor so that certain keywords always rank at the top, speech model testing across regional accents, and a display that shows the ranking confidence. The vendor’s medical director accepts the residual risk in writing. Each quarter the file is updated with incident data from the dispatch centers that run the tool.
What to watch for
- A one-time risk assessment at launch does not satisfy the Act, which asks for a process that keeps running.
- Define the metrics and thresholds you will test against before testing, because thresholds chosen afterwards look like rationalization.
- Include foreseeable misuse, such as deployers using the system outside the documented population or purpose.
- Feed post-market monitoring data back into the risk file, since that loop is what auditors check first.