What is an AI audit?
An AI audit is a formal examination of an AI system, its data, and the process around it, carried out against stated criteria and backed by evidence. The criteria might be internal policy, a framework such as the NIST AI Risk Management Framework or ISO/IEC 42001, a regulation, or a customer’s contract. The auditor collects documents, inspects logs, re-runs tests, and interviews the people who built and operate the system, then reports findings and required fixes.
Audits differ from monitoring. Model monitoring runs continuously and watches for drift and errors. An audit happens at a point in time, covers the whole system including the human process, and produces an independent opinion. Internal audit teams, external firms, and regulators all perform AI audits, and a conformity assessment under the EU AI Act is one specific form.
Why an AI audit matters
An audit is how a company proves, rather than asserts, that an AI system is under control. Companies that cannot produce that proof face delayed deals with large customers and findings in their SOC 2 or financial audits. For AI agents that change records in business systems, the audit also confirms that approval steps and access limits are enforced in practice and were not switched off during a busy period.
Getting started with an AI audit
Planning for an audit begins long before the auditor arrives. A regional health system that runs a model to flag insurance claims for review, plus an agent that drafts appeal letters, prepares in four steps. First, it keeps an AI inventory listing each system, its owner, its purpose, and its risk tier. Second, it maintains the evidence an auditor will request: model cards, training data descriptions, test results, fairness checks, change records, and the AI audit log of model calls and agent actions. Third, it maps each control to the framework the audit will use. Fourth, it runs a self-assessment each quarter so that findings surface internally first.
In Kirk Tech Solutions engagements, audit readiness is designed in from the start: every agent action is logged with the user, the model version, the inputs, and the approver, so the evidence exists without a separate collection effort.
What to watch for
- Agree on the audit criteria in writing before fieldwork, because an audit against vague expectations produces vague findings.
- Make sure logs cannot be edited by the teams being audited, or their value as evidence drops.
- Include the people and the process, such as who approves changes and how incidents are handled, since many failures come from process gaps rather than from the model.
- Audit agents as well as models, checking that tool permissions and approval steps match the design documents.
- Plan time and budget for fixing the findings and for a follow-up check that confirms they were fixed.