# A Framework for Assurance Audits of Algorithmic Systems

arXiv: https://arxiv.org/abs/2401.14908 · HTML: https://mlsanity.com/paper/2401.14908/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2024.

## Built on by (most cited first, 8 of 8)

- [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024)
- [Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems](https://mlsanity.com/paper/2605.16198/) (2026)
- [With a Little Help From My Friends: Collective Manipulation in Risk-Controlling Recommender Systems](https://mlsanity.com/paper/2603.28476/) (2026)
- [The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing](https://mlsanity.com/paper/2601.11783/) (2026)
- [Trustworthy AI Posture (TAIP): A Framework for Continuous AI Assurance of Agentic Systems at Horizontal and Vertical scale](https://mlsanity.com/paper/2603.03340/) (2026)
- [Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software](https://mlsanity.com/paper/2604.10875/) (2026)
- [Efficient Fairness Auditing Across Guidance Scales in Text-to-Image Diffusion Models via Causal Abstraction](https://mlsanity.com/paper/2609.09486/) (2026)
- [Who Audits Whom, on What Substrate, with What Evidence? An Independence-Graded Audit Protocol for Agentic AI](https://mlsanity.com/paper/2609.18272/) (2026)

## Abstract (arXiv)

An increasing number of regulations propose AI audits as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently lacks agreed-upon practices, procedures, taxonomies, and standards. We propose the criterion audit as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations' ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which bias audits can be performed for in-scope hiring algorithms, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion -- informed by experiences in performing these audits in practice -- highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of audits.