# Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

arXiv: https://arxiv.org/abs/2402.17861 · HTML: https://mlsanity.com/paper/2402.17861/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2024.

## Built on by (most cited first, 29 of 29)

- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It](https://mlsanity.com/paper/2505.24119/) (2025)
- [Responsible Reporting for Frontier AI Development](https://mlsanity.com/paper/2404.02675/) (2024)
- [A Framework for Responsible AI Systems: Building Societal Trust through Domain Definition, Trustworthy AI Design, Auditability, Accountability, and Governance](https://mlsanity.com/paper/2503.04739/) (2025)
- [Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms](https://mlsanity.com/paper/2510.10339/) (2025)
- [Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms](https://mlsanity.com/paper/2605.30169/) (2026)
- [Sound Check: Auditing Audio Datasets](https://mlsanity.com/paper/2410.13114/) (2024)
- [A Comprehensive Survey and Classification of Evaluation Criteria for Trustworthy Artificial Intelligence](https://mlsanity.com/paper/2410.17281/) (2024)
- [From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms](https://mlsanity.com/paper/2502.08166/) (2025)
- [Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems](https://mlsanity.com/paper/2506.04482/) (2025)
- [The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don't](https://mlsanity.com/paper/2605.01771/) (2026)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [Ethical AI Governance: Methods for Evaluating Trustworthy AI](https://mlsanity.com/paper/2409.07473/) (2024)
- [The Hall of AI Fears and Hopes: Comparing the Views of AI Influencers and those of Members of the U.S. Public Through an Interactive Platform](https://mlsanity.com/paper/2504.06016/) (2025)
- [What Does 'Human-Centred AI' Mean?](https://mlsanity.com/paper/2507.19960/) (2025)
- [Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives](https://mlsanity.com/paper/2509.05627/) (2025)
- [FairFare: A Tool for Crowdsourcing Rideshare Data to Empower Labor Organizers](https://mlsanity.com/paper/2502.11273/) (2025)
- [PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluation](https://mlsanity.com/paper/2601.11702/) (2026)
- [Trustworthy AI Posture (TAIP): A Framework for Continuous AI Assurance of Agentic Systems at Horizontal and Vertical scale](https://mlsanity.com/paper/2603.03340/) (2026)
- [Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems](https://mlsanity.com/paper/2505.19441/) (2025)
- [TVS Sidekick: Challenges and Practical Insights from Deploying Large Language Models in the Enterprise](https://mlsanity.com/paper/2509.26482/) (2025)
- [Can AI be Accountable?](https://mlsanity.com/paper/2510.26057/) (2025)
- [Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software](https://mlsanity.com/paper/2604.10875/) (2026)
- [Artificial intelligence language technologies in multilingual healthcare: Grand challenges ahead](https://mlsanity.com/paper/2605.01441/) (2026)
- [Closing the AI Trust Gap: The Case for Independent Certification for Trustworthy AI](https://mlsanity.com/paper/2607.15992/) (2026)
- [The Acknowledgment Point Is the System: Durable Policy-Decision Receipts for AI Audit Evidence](https://mlsanity.com/paper/2608.17176/) (2026)
- [SoK: Formal Methods for Fact-Checking and Information Integrity](https://mlsanity.com/paper/2609.23239/) (2026)
- [Actionable Meta-Requirements for AI Robustness Auditing](https://mlsanity.com/paper/2610.14356/) (2026)

## Abstract (arXiv)

Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs. While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability. We outline challenges practitioners faced in their efforts to use AI audit tools and highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy. We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the field move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.