mlsanityData to 7 Oct 2026

Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

arXiv 2001.00973 ↗First posted 3 Jan 2020cs.CYcited by 380

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source. In this paper, we introduce a framework for algorithmic auditing that supports artificial intelligence system development end-to-end, to be applied throughout the internal organization development lifecycle. Each stage of the audit yields a set of documents that together form an overall audit report, drawing on an organization's values or principles to assess the fit of decisions made throughout the process. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale artificial intelligence systems by embedding a robust process to ensure audit integrity.

Key facts. Cited by 380 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 45; 2023: 59; 2024: 51; 2025: 53; 2026: 136.

380 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2021Trustworthy AI: From Principles to Practicescited by 206
  4. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  5. 2023Model evaluation for extreme riskscited by 167
  6. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  7. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  8. 2023Auditing large language models: a three-layered approachcited by 135
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  11. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  12. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
38 more
  1. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  2. 2021Trustworthy AI: A Computational Perspectivecited by 93
  3. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  4. 2022The Fallacy of AI Functionalitycited by 88
  5. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  6. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  7. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  8. 2021Software Engineering for AI-Based Systems: A Surveycited by 68
  9. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  10. 2023The Gradient of Generative AI Release: Methods and Considerationscited by 63
  11. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 61
  12. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  13. 2022Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learningcited by 53
  14. 2022How Different Groups Prioritize Ethical Values for Responsible AIcited by 49
  15. 2021Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffscited by 48
  16. 2022Designing for Responsible Trust in AI Systems: A Communication Perspectivecited by 48
  17. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  18. 2022Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learningcited by 47
  19. 2023Supporting Human-AI Collaboration in Auditing LLMs with LLMscited by 46
  20. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  21. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  22. 2025Infrastructure for AI Agentscited by 40
  23. 2023Understanding accountability in algorithmic supply chainscited by 39
  24. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  25. 2022A Comprehensive Survey on Trustworthy Recommender Systemscited by 32
  26. 2024What are human values, and how do we align AI to them?cited by 32
  27. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  28. 2021Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systemscited by 30
  29. 2022A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithmscited by 30
  30. 2022Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practicecited by 30
  31. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  32. 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
  33. 2024Identifying Implicit Social Biases in Vision-Language Modelscited by 27
  34. 2021Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systemscited by 26
  35. 2022The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligencecited by 26
  36. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 26
  37. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  38. 2023Uncovering and Quantifying Social Biases in Code Generationcited by 25

50 most cited of 380 All 380 (JSON)