Human oversight
Human oversight requires high-risk AI systems to be designed so they can be effectively supervised by natural persons during their use. The objective is to prevent or minimize risks to health, safety, or fundamental rights by ensuring humans can intervene, override, or shut down the system.
What it means
The requirement moves beyond simply having a "human in the loop." It mandates that oversight must be *effective*, meaning the person supervising the AI must have the technical competence, authority, and actual ability to counteract the system's output.
In practice, this means addressing "automation bias"—the tendency for humans to over-rely on automated suggestions without critical evaluation. The design of the AI must actively discourage blind reliance and instead facilitate a critical human review process.
The scope applies specifically to high-risk AI systems. Depending on the system's function, oversight may be implemented as a constant real-time monitor or through periodic reviews of the system's performance and decision logs.
How to meet it
- Build "stop" or "override" mechanisms into the user interface that allow an overseer to immediately disregard or reverse AI outputs.
- Create detailed operational manuals that explicitly define when a human must intervene and how to do so.
- Develop a training program for overseeers that covers the system's limitations, known biases, and the risks of automation bias.
- Define clear organizational roles and responsibilities, designating exactly which personnel are responsible for oversight tasks.
- Ensure AI outputs are presented in an interpretable format (explainability) so the human overseer can make a reasoned decision to override them.
- Establish a protocol for logging all instances where a human intervened or overrode the system to allow for future risk assessment.
Evidence an auditor asks for
- Technical documentation describing the Human-Machine Interface (HMI) and the specific mechanisms used for overrides or shutdowns.
- Training logs and certificates proving that designated overseers have been trained on the AI's limitations and oversight procedures.
- A formal "Human Oversight Plan" or Standard Operating Procedure (SOP) detailing triggers for intervention and escalation paths.
- Audit trails or system logs showing historical instances where human overseers overrode, corrected, or ignored AI-generated outputs.
Common pitfalls
- Rubber stamping: Implementing a process where humans simply click "approve" on all AI outputs without performing an actual review.
- Competence gaps: Assigning oversight to staff who lack the domain expertise to recognize when an AI output is incorrect or biased.
- Poor interface design: Creating a system where the override function is buried in menus, making it impossible to intervene in time to prevent harm.