# Auditing of AI: Legal, Ethical and Technical Approaches arXiv: https://arxiv.org/abs/2407.06235 ยท HTML: https://mlsanity.com/paper/2407.06235/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 62 of its 208 references that are in the map. - First posted on arXiv 7 Jul 2024. ## Builds on (most cited first, 30 of 62) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022) - [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020) - [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022) - [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017) - [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018) - [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019) - [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017) - [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020) - [On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning](https://mlsanity.com/paper/2110.11891/) (2021) - [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023) - [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020) - [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018) - [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021) - [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023) - [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021) - [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021) - [Artificial Intelligence: the global landscape of ethics guidelines](https://mlsanity.com/paper/1906.11668/) (2019) - [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018) - [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018) - [Discrimination in Online Ad Delivery](https://mlsanity.com/paper/1301.6822/) (2013) - [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019) - [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022) - [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022) - [Demystifying the Draft EU Artificial Intelligence Act](https://mlsanity.com/paper/2107.03721/) (2021) - [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016) - [Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem](https://mlsanity.com/paper/2310.02521/) (2023) - [Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors](https://mlsanity.com/paper/2105.02980/) (2021) ## Built on by (most cited first, 10 of 10) - [Attestable Audits: Verifiable AI Safety Benchmarks Using Trusted Execution Environments](https://mlsanity.com/paper/2506.23706/) (2025) - [Embodied AI: Emerging Risks and Opportunities for Policy Action](https://mlsanity.com/paper/2509.00117/) (2025) - [Assessing the Reliability of Large Language Models in the Bengali Legal Context: A Comparative Evaluation Using LLM-as-Judge and Legal Experts](https://mlsanity.com/paper/2511.05627/) (2025) - [PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluation](https://mlsanity.com/paper/2601.11702/) (2026) - [Who Owns The Robot?: Four Ethical and Socio-technical Questions about Wellbeing Robots in the Real World through Community Engagement](https://mlsanity.com/paper/2509.02624/) (2025) - [Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Space](https://mlsanity.com/paper/2608.07786/) (2026) - [Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level](https://mlsanity.com/paper/2608.14692/) (2026) - [MARLA: A Conceptual Scaffold for Regulatory Learning under the EU AI Act](https://mlsanity.com/paper/2609.04877/) (2026) - [The AI Assessment Sandbox Configurator: A Framework to Support Technical Assessment in AI Regulatory Sandboxes](https://mlsanity.com/paper/2610.01539/) (2026) - [Actionable Meta-Requirements for AI Robustness Auditing](https://mlsanity.com/paper/2610.14356/) (2026) ## Abstract (arXiv) AI auditing is a rapidly growing field of research and practice. This review article, which doubles as an editorial to Digital Societys topical collection on Auditing of AI, provides an overview of previous work in the field. Three key points emerge from the review. First, contemporary attempts to audit AI systems have much to learn from how audits have historically been structured and conducted in areas like financial accounting, safety engineering and the social sciences. Second, both policymakers and technology providers have an interest in promoting auditing as an AI governance mechanism. Academic researchers can thus fill an important role by studying the feasibility and effectiveness of different AI auditing procedures. Third, AI auditing is an inherently multidisciplinary undertaking, to which substantial contributions have been made by computer scientists and engineers as well as social scientists, philosophers, legal scholars and industry practitioners. Reflecting this diversity of perspectives, different approaches to AI auditing have different affordances and constraints. Specifically, a distinction can be made between technology-oriented audits, which focus on the properties and capabilities of AI systems, and process oriented audits, which focus on technology providers governance structures and quality management systems. The next step in the evolution of auditing as an AI governance mechanism, this article concludes, should be the interlinking of these available (and complementary) approaches into structured and holistic procedures to audit not only how AI systems are designed and used but also how they impact users, societies and the natural environment in applied settings over time.