What is a high-risk AI system?
A high-risk AI system is an AI system that the EU AI Act places in its most heavily regulated permitted tier because of its potential effect on health, safety, or fundamental rights. The Act defines the tier in two ways. Annex I covers AI that is a safety component of a product already governed by EU product legislation, such as medical devices, elevators, pressure equipment, and vehicles. AI safety functions in machinery follow the same logic but are regulated through the EU Machinery Regulation. Annex III lists use areas where AI decides or heavily influences outcomes for people: biometrics, critical infrastructure, education, employment and worker management, access to essential services including credit and insurance, law enforcement, migration, and the administration of justice.
The label attaches to the system in its intended use. A scoring model is minimal risk when it ranks warehouse replenishment orders and becomes high-risk when the same technique ranks insurance applicants.
How high-risk AI system obligations work
The provider, meaning the company that builds the system or sells it under its own name, carries most of the weight. It must run a documented risk management system, govern the training and testing data, write technical documentation, build in automatic logging, give deployers instructions for use, design for human oversight, and meet requirements for accuracy, robustness, and cybersecurity. It must pass a conformity assessment, affix the CE marking, register the system in the EU database, and run post-market monitoring with serious incident reporting.
The deployer, the company using the system, has a shorter but real list. It must use the system according to the instructions, assign oversight to trained people with the authority to intervene, make sure input data is relevant, keep the logs the system generates for at least six months, inform workers and their representatives when the system affects them, inform individuals subject to its decisions, and in some cases complete a fundamental rights impact assessment.
High-risk AI systems in practice
A medical device maker adds a model that flags suspected fractures on X-ray images for the radiologist. Because the software is itself a medical device that needs notified-body assessment, it is high-risk under Annex I, and the maker’s existing conformity process expands to cover the model’s documentation, testing evidence, and logging. A hospital that buys the software has deployer duties only: follow the instructions, assign trained staff to oversee it, keep the logs, and report incidents.
What to watch for
- Changing a system’s purpose or making a substantial modification can turn a deployer into a provider with the full obligation set.
- Instructions for use are a legal document, so read them before deployment and respect the operating limits they set.
- Human oversight has to be real, meaning the overseer has the time, training, and authority to overrule the system.
- Collect the provider’s documentation at procurement, since obtaining it after signature is harder.