Monitoring, internal audit and management review
Clause 9 requires the organization to evaluate the performance of its Artificial Intelligence Management System (AIMS). This involves continuous monitoring of AI system effectiveness, conducting objective internal audits for compliance, and holding formal management reviews to ensure the system remains suitable and effective.
What it means
In practice, this clause is about "closing the loop" in the Plan-Do-Check-Act cycle. It ensures that you are not just implementing policies, but actively verifying that those policies work and that your AI systems behave as intended. Monitoring focuses on metrics and KPIs, while auditing focuses on adherence to the standard and internal rules.
Management review is the governance layer. It requires senior leadership to step back from daily operations and examine the AIMS holistically. The goal is to determine if the system needs strategic adjustments based on audit results, changing risk landscapes, or shifts in organizational goals.
How to meet it
- Define specific metrics for monitoring AI performance and AIMS effectiveness (e.g., tracking AI bias incidents, accuracy drift, or policy compliance rates).
- Establish a formal internal audit program that defines the frequency, methods, and responsibilities for auditing different parts of the AIMS.
- Ensure auditors are objective and impartial; they should not audit their own work or the systems they personally manage.
- Conduct scheduled management review meetings with an agenda covering audit results, feedback from interested parties, and status of risk treatments.
- Document a clear process for how monitoring data and audit findings are escalated to leadership for decision-making.
- Create a loop where outputs from these reviews lead directly into corrective actions (Clause 10).
Evidence an auditor asks for
- Monitoring reports or dashboards demonstrating that AI systems and management controls are being tracked against defined KPIs.
- An internal audit schedule/plan and the corresponding detailed audit reports identifying non-conformities.
- Minutes of management review meetings, including a record of attendees and decisions made.
- Records of corrective actions triggered by an internal audit or a management review finding.
Common pitfalls
- Confusing technical AI monitoring (e.g., model accuracy) with AIMS monitoring (e.g., whether the risk management process is being followed).
- Performing "paper audits" where checklists are marked as complete without actual evidence verification.
- Treating the management review as a status update rather than a critical evaluation of the system's effectiveness and suitability.