# Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem arXiv: https://arxiv.org/abs/2310.02521 ยท HTML: https://mlsanity.com/paper/2310.02521/ ## Key facts - Cited by 49 papers in the map (data through 7 Oct 2026). - First posted on arXiv 4 Oct 2023. ## Built on by (most cited first, 30 of 49) - [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024) - [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024) - [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024) - [Infrastructure for AI Agents](https://mlsanity.com/paper/2501.10114/) (2025) - [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025) - [HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router](https://mlsanity.com/paper/2410.02684/) (2024) - [From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing](https://mlsanity.com/paper/2410.04772/) (2024) - [The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence](https://mlsanity.com/paper/2408.12622/) (2024) - [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024) - [Auditing of AI: Legal, Ethical and Technical Approaches](https://mlsanity.com/paper/2407.06235/) (2024) - [Addressing the regulatory gap: moving towards an EU AI audit ecosystem beyond the AI Act by including civil society](https://mlsanity.com/paper/2403.07904/) (2024) - [Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis](https://mlsanity.com/paper/2410.01817/) (2024) - [In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI](https://mlsanity.com/paper/2503.16861/) (2025) - [Robust ML Auditing using Prior Knowledge](https://mlsanity.com/paper/2505.04796/) (2025) - [An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI](https://mlsanity.com/paper/2310.14455/) (2023) - [Law and the Emerging Political Economy of Algorithmic Audits](https://mlsanity.com/paper/2406.11855/) (2024) - [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025) - [Accountability Capture: How Record-Keeping to Support AI Transparency and Accountability (Re)shapes Algorithmic Oversight](https://mlsanity.com/paper/2510.04609/) (2025) - [Addressing Discretization-Induced Bias in Demographic Prediction](https://mlsanity.com/paper/2405.16762/) (2024) - [Data Ethics Emergency Drill: A Toolbox for Discussing Responsible AI for Industry Teams](https://mlsanity.com/paper/2403.10438/) (2024) - [(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court](https://mlsanity.com/paper/2403.13004/) (2024) - [Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs](https://mlsanity.com/paper/2411.04994/) (2024) - ["Unlimited Realm of Exploration and Experimentation": Methods and Motivations of AI-Generated Sexual Content Creators](https://mlsanity.com/paper/2601.21028/) (2026) - [Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems](https://mlsanity.com/paper/2605.16198/) (2026) - [Assessing AI Impact Assessments: A Classroom Study](https://mlsanity.com/paper/2311.11193/) (2023) - [Making Data Work Count](https://mlsanity.com/paper/2311.18046/) (2023) - [Rolling in the deep of cognitive and AI biases](https://mlsanity.com/paper/2407.21202/) (2024) - [Interpretable and Fair Mechanisms for Abstaining Classifiers](https://mlsanity.com/paper/2503.18826/) (2025) - [What Is AI Safety? What Do We Want It to Be?](https://mlsanity.com/paper/2505.02313/) (2025) - [Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives](https://mlsanity.com/paper/2509.05627/) (2025) ## Abstract (arXiv) AI audits are an increasingly popular mechanism for algorithmic accountability; however, they remain poorly defined. Without a clear understanding of audit practices, let alone widely used standards or regulatory guidance, claims that an AI product or system has been audited, whether by first-, second-, or third-party auditors, are difficult to verify and may exacerbate, rather than mitigate, bias and harm. To address this knowledge gap, we provide the first comprehensive field scan of the AI audit ecosystem. We share a catalog of individuals (N=438) and organizations (N=189) who engage in algorithmic audits or whose work is directly relevant to algorithmic audits; conduct an anonymous survey of the group (N=152); and interview industry leaders (N=10). We identify emerging best practices as well as methods and tools that are becoming commonplace, and enumerate common barriers to leveraging algorithmic audits as effective accountability mechanisms. We outline policy recommendations to improve the quality and impact of these audits, and highlight proposals with wide support from algorithmic auditors as well as areas of debate. Our recommendations have implications for lawmakers, regulators, internal company policymakers, and standards-setting bodies, as well as for auditors. They are: 1) require the owners and operators of AI systems to engage in independent algorithmic audits against clearly defined standards; 2) notify individuals when they are subject to algorithmic decision-making systems; 3) mandate disclosure of key components of audit findings for peer review; 4) consider real-world harm in the audit process, including through standardized harm incident reporting and response mechanisms; 5) directly involve the stakeholders most likely to be harmed by AI systems in the algorithmic audit process; and 6) formalize evaluation and, potentially, accreditation of algorithmic auditors.