# The Necessity of AI Audit Standards Boards

arXiv: https://arxiv.org/abs/2404.13060 · HTML: https://mlsanity.com/paper/2404.13060/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 98 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 29 of 29)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B](https://mlsanity.com/paper/2310.20624/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance](https://mlsanity.com/paper/2206.04737/) (2022)
- [Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools](https://mlsanity.com/paper/2306.13952/) (2023)
- [AI auditing: The Broken Bus on the Road to AI Accountability](https://mlsanity.com/paper/2401.14462/) (2024)
- [BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B](https://mlsanity.com/paper/2311.00117/) (2023)
- [The Data-Production Dispositif](https://mlsanity.com/paper/2205.11963/) (2022)
- [Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Framework](https://mlsanity.com/paper/2311.14711/) (2023)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [TrISec: Training Data-Unaware Imperceptible Security Attacks on Deep Neural Networks](https://mlsanity.com/paper/1811.01031/) (2018)
- [Predictable Artificial Intelligence](https://mlsanity.com/paper/2310.06167/) (2023)
- [An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI](https://mlsanity.com/paper/2310.14455/) (2023)
- [Audit and Assurance of AI Algorithms: A framework to ensure ethical algorithmic practices in Artificial Intelligence](https://mlsanity.com/paper/2107.14046/) (2021)
- [Frontier AI developers need an internal audit function](https://mlsanity.com/paper/2305.17038/) (2023)
- [Safeguarding the safeguards: How best to promote AI alignment in the public interest](https://mlsanity.com/paper/2312.08039/) (2023)

## Built on by (most cited first, 9 of 9)

- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [AI Governance through Markets](https://mlsanity.com/paper/2501.17755/) (2025)
- [Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance](https://mlsanity.com/paper/2503.04736/) (2025)
- [A Framework for Responsible AI Systems: Building Societal Trust through Domain Definition, Trustworthy AI Design, Auditability, Accountability, and Governance](https://mlsanity.com/paper/2503.04739/) (2025)
- [Limits of Safe AI Deployment: Differentiating Oversight and Control](https://mlsanity.com/paper/2507.03525/) (2025)
- [Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment](https://mlsanity.com/paper/2512.04210/) (2025)
- [Reciprocal Trust and Distrust in Artificial Intelligence Systems: The Hard Problem of Regulation](https://mlsanity.com/paper/2604.05826/) (2026)
- [The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance](https://mlsanity.com/paper/2606.28710/) (2026)
- [AISPA: User-Centric System Prompt Auditing for Large Language Model Applications](https://mlsanity.com/paper/2607.28617/) (2026)

## Abstract (arXiv)

Auditing of AI systems is a promising way to understand and manage ethical problems and societal risks associated with contemporary AI systems, as well as some anticipated future risks. Efforts to develop standards for auditing Artificial Intelligence (AI) systems have therefore understandably gained momentum. However, we argue that creating auditing standards is not just insufficient, but actively harmful by proliferating unheeded and inconsistent standards, especially in light of the rapid evolution and ethical and safety challenges of AI. Instead, the paper proposes the establishment of an AI Audit Standards Board, responsible for developing and updating auditing methods and standards in line with the evolving nature of AI technologies. Such a body would ensure that auditing practices remain relevant, robust, and responsive to the rapid advancements in AI. The paper argues that such a governance structure would also be helpful for maintaining public trust in AI and for promoting a culture of safety and ethical responsibility within the AI industry. Throughout the paper, we draw parallels with other industries, including safety-critical industries like aviation and nuclear energy, as well as more prosaic ones such as financial accounting and pharmaceuticals. AI auditing should emulate those fields, and extend beyond technical assessments to include ethical considerations and stakeholder engagement, but we explain that this is not enough; emulating other fields' governance mechanisms for these processes, and for audit standards creation, is a necessity. We also emphasize the importance of auditing the entire development process of AI systems, not just the final products...