Accuracy, robustness and cybersecurity

Article 15 requires high-risk AI systems to be designed and developed to achieve appropriate levels of accuracy, robustness, and cybersecurity throughout their lifecycle. This means the system must perform consistently as intended, resist errors or perturbations, and remain resilient against unauthorized third-party attempts to alter its behavior or data.

What it means

The intent is to prevent high-risk AI from causing harm due to technical failure or malicious interference. Rather than prescribing a universal percentage for "accuracy," the Act requires providers to determine what level of performance is appropriate based on the specific intended purpose and the risks associated with that use case.

Robustness focuses on the system's ability to maintain its level of performance even when facing unexpected inputs, noise, or environmental changes. It ensures the AI does not produce erratic or dangerous outputs when encountering "edge cases" that were not prevalent in the training data.

Cybersecurity extends beyond standard IT security. In this context, it specifically addresses vulnerabilities unique to AI, such as adversarial attacks (inputs designed to fool a model), data poisoning during training, and prompt injection, ensuring these cannot compromise the system's safety or fundamental rights.

How to meet it

Define quantitative metrics for accuracy, robustness, and cybersecurity that are tailored to the specific use case and risk profile of the AI system.
Establish a rigorous testing framework that includes stress tests and "edge-case" scenarios to verify performance under adverse conditions.
Implement adversarial testing (red teaming) to identify vulnerabilities where malicious inputs could cause the model to malfunction or leak data.
Integrate security controls throughout the development pipeline, including secure data sourcing to prevent poisoning and encrypted communication channels for deployment.
Create a continuous monitoring plan to detect "model drift" or performance degradation over time, ensuring accuracy is maintained post-deployment.
Document the rationale behind why chosen performance thresholds were deemed "appropriate" for the given application.

Evidence an auditor asks for

Technical documentation specifying the exact metrics used to measure accuracy and robustness and the justification for these benchmarks.
Validation and verification reports showing test results against those predefined metrics, including failures and subsequent remediations.
A cybersecurity risk assessment that identifies AI-specific threats and documents the controls implemented to mitigate them.
Logs of adversarial testing or penetration tests conducted on the model before release.
Evidence of a quality management system (QMS) that tracks versioning and performance updates throughout the lifecycle.

Common pitfalls

Relying on generic industry benchmarks instead of defining metrics specific to the actual operational environment of the AI.
Confusing general corporate IT security (e.g., firewalls, passwords) with AI-specific cybersecurity requirements like resilience against adversarial perturbations.
Treating accuracy and robustness as a "one-time" checkbox at launch rather than an ongoing requirement that must be monitored during the system's entire operational life.
Failing to document the decision-making process used to determine what constitutes an "appropriate" level of performance.