mlsanityData to 14 Oct 2026

AI auditing: The Broken Bus on the Road to AI Accountability

arXiv 2401.14462 ↗First posted 25 Jan 2024cs.CYcited by 43

Not on the map. Why?

Abstract (arXiv)

One of the most concrete measures to take towards meaningful AI accountability is to consequentially assess and report the systems' performance and impact. However, the practical nature of the "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. Next, we assess the impact of audits done by stakeholders within each domain. We find that only a subset of AI audit studies translate to desired accountability outcomes. We thus assess and isolate practices necessary for effective AI audit results, articulating the observed connections between AI audit design, methodology and institutional context on its effectiveness as a meaningful mechanism for accountability.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Jan 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 15; 2026: 15.

43 papers.

Sort:

By area:

  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  2. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  3. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  4. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  5. 2024The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiringcited by 29
  6. 2024The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligencecited by 23
  7. 2025Lessons From Red Teaming 100 Generative AI Productscited by 23
  8. 2026Auditable Agentscited by 15
  9. 2025Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variationscited by 13
  10. 2024Responsible Reporting for Frontier AI Developmentcited by 10
  11. 2024How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studiescited by 10
  12. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
31 more
  1. 2024The Necessity of AI Audit Standards Boardscited by 9
  2. 2024Addressing the regulatory gap: moving towards an EU AI audit ecosystem beyond the AI Act by including civil societycited by 7
  3. 2025The Accountability Paradox: How Platform API Restrictions Undermine AI Transparency Mandatescited by 7
  4. 2025Robust ML Auditing using Prior Knowledgecited by 6
  5. 2026Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMscited by 6
  6. 2024To Err is AI : A Case Study Informing LLM Flaw Reporting Practicescited by 5
  7. 2025Accountability Capture: How Record-Keeping to Support AI Transparency and Accountability (Re)shapes Algorithmic Oversightcited by 5
  8. 2026Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filterscited by 5
  9. 2026Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanismscited by 4
  10. 2024Watching the Watchers: A Comparative Fairness Audit of Cloud-based Content Moderation Servicescited by 3
  11. 2024UTrace: Poisoning Forensics for Private Collaborative Learningcited by 3
  12. 2024Sound Check: Auditing Audio Datasetscited by 3
  13. 2025Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Frameworkcited by 3
  14. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lenscited by 3
  15. 2026Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systemscited by 3
  16. 2024SoK: Security and Privacy Risks of Healthcare AIcited by 2
  17. 2025P2NIA: Privacy-Preserving Non-Iterative Auditingcited by 2
  18. 2024Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AIcited by 1
  19. 2025AI Agent Architecture for Decentralized Trading of Alternative Assetscited by 1
  20. 2026MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loopcited by 1
  21. 2025Can AI be Accountable?cited by 0
  22. 2025Seeing Twice: How Side-by-Side T2I Comparison Changes Auditing Strategiescited by 0
  23. 2026Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Softwarecited by 0
  24. 2026PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AIcited by 0
  25. 2026Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Researchcited by 0
  26. 2026Closing the AI Trust Gap: The Case for Independent Certification for Trustworthy AIcited by 0
  27. 2026AISPA: User-Centric System Prompt Auditing for Large Language Model Applicationscited by 0
  28. 2026A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AIcited by 0
  29. 2026Follow the Norm: Accounting for Fine-Tuning and Prompt Effects on Model Rationalescited by 0
  30. 2026Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Levelcited by 0
  31. 2026VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Predictioncited by 0