mlsanityData to 7 Oct 2026

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

arXiv 2402.17861 ↗First posted 27 Feb 2024cs.CYcited by 27

Not on the map. Why?

Abstract (arXiv)

Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs. While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability. We outline challenges practitioners faced in their efforts to use AI audit tools and highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy. We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the field move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 8.

27 papers.

Sort:

By area:

  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  2. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  3. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  4. 2024Responsible Reporting for Frontier AI Developmentcited by 10
  5. 2025A Framework for Responsible AI Systems: Building Societal Trust through Domain Definition, Trustworthy AI Design, Auditability, Accountability, and Governancecited by 5
  6. 2025Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harmscited by 5
  7. 2024Sound Check: Auditing Audio Datasetscited by 3
  8. 2024A Comprehensive Survey and Classification of Evaluation Criteria for Trustworthy Artificial Intelligencecited by 3
  9. 2025From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harmscited by 3
  10. 2025Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systemscited by 3
  11. 2026Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanismscited by 3
  12. 2024A Taxonomy of Challenges to Curating Fair Datasetscited by 2
15 more
  1. 2024Ethical AI Governance: Methods for Evaluating Trustworthy AIcited by 2
  2. 2025The Hall of AI Fears and Hopes: Comparing the Views of AI Influencers and those of Members of the U.S. Public Through an Interactive Platformcited by 2
  3. 2025What Does 'Human-Centred AI' Mean?cited by 2
  4. 2025Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternativescited by 2
  5. 2025FairFare: A Tool for Crowdsourcing Rideshare Data to Empower Labor Organizerscited by 1
  6. 2026PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluationcited by 1
  7. 2026Trustworthy AI Posture (TAIP): A Framework for Continuous AI Assurance of Agentic Systems at Horizontal and Vertical scalecited by 1
  8. 2025Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systemscited by 0
  9. 2025TVS Sidekick: Challenges and Practical Insights from Deploying Large Language Models in the Enterprisecited by 0
  10. 2025Can AI be Accountable?cited by 0
  11. 2026Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Softwarecited by 0
  12. 2026Artificial intelligence language technologies in multilingual healthcare: Grand challenges aheadcited by 0
  13. 2026Closing the AI Trust Gap: The Case for Independent Certification for Trustworthy AIcited by 0
  14. 2026The Acknowledgment Point Is the System: Durable Policy-Decision Receipts for AI Audit Evidencecited by 0
  15. 2026SoK: Formal Methods for Fact-Checking and Information Integritycited by 0