# Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies

arXiv: https://arxiv.org/abs/2109.06974 · HTML: https://mlsanity.com/paper/2109.06974/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2021.

## Built on by (most cited first, 22 of 22)

- [Evaluating and Mitigating Discrimination in Language Model Decisions](https://mlsanity.com/paper/2312.03689/) (2023)
- [What's in a Name? Auditing Large Language Models for Race and Gender Bias](https://mlsanity.com/paper/2402.14875/) (2024)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring](https://mlsanity.com/paper/2201.09151/) (2022)
- [Auditing of AI: Legal, Ethical and Technical Approaches](https://mlsanity.com/paper/2407.06235/) (2024)
- [CALM: Curiosity-Driven Auditing for Large Language Models](https://mlsanity.com/paper/2501.02997/) (2025)
- [Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study](https://mlsanity.com/paper/2305.18307/) (2023)
- [Leave no Place Behind: Improved Geolocation in Humanitarian Documents](https://mlsanity.com/paper/2309.02914/) (2023)
- [Racial/Ethnic Categories in AI and Algorithmic Fairness: Why They Matter and What They Represent](https://mlsanity.com/paper/2404.06717/) (2024)
- [Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers](https://mlsanity.com/paper/2410.22937/) (2024)
- [Kantian Deontology Meets AI Alignment: Towards Morally Grounded Fairness Metrics](https://mlsanity.com/paper/2311.05227/) (2023)
- [Auditing Fairness under Model Updates: Fundamental Complexity and Property-Preserving Updates](https://mlsanity.com/paper/2601.05909/) (2026)
- [Global Sequential Testing for Multi-Stream Auditing](https://mlsanity.com/paper/2602.21479/) (2026)
- [Manipulation-Proof Oblivious Audits against Deceptive Model Providers](https://mlsanity.com/paper/2608.04365/) (2026)
- [Strategic Evaluation: Subjects, Evaluators, and Society](https://mlsanity.com/paper/2310.03655/) (2023)
- [Why Algorithms Remain Unjust: Power Structures Surrounding Algorithmic Activity](https://mlsanity.com/paper/2405.18461/) (2024)
- [Testing software for non-discrimination: an updated and extended audit in the Italian car insurance domain](https://mlsanity.com/paper/2502.06439/) (2025)
- [The Illusion of Fairness: Auditing Fairness Interventions with Audit Studies](https://mlsanity.com/paper/2507.02152/) (2025)
- [A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI](https://mlsanity.com/paper/2608.04921/) (2026)
- [Small-world Networks of Agents Brainstorm AI Risks to Support Ideation](https://mlsanity.com/paper/2609.24859/) (2026)

## Abstract (arXiv)

Algorithmic audits have been embraced as tools to investigate the functioning and consequences of sociotechnical systems. Though the term is used somewhat loosely in the algorithmic context and encompasses a variety of methods, it maintains a close connection to audit studies in the social sciences--which have, for decades, used experimental methods to measure the prevalence of discrimination across domains like housing and employment. In the social sciences, audit studies originated in a strong tradition of social justice and participatory action, often involving collaboration between researchers and communities; but scholars have argued that, over time, social science audits have become somewhat distanced from these original goals and priorities. We draw from this history in order to highlight difficult tensions that have shaped the development of social science audits, and to assess their implications in the context of algorithmic auditing. In doing so, we put forth considerations to assist in the development of robust and engaged assessments of sociotechnical systems that draw from auditing's roots in racial equity and social justice.