# Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice

arXiv: https://arxiv.org/abs/2210.03709 · HTML: https://mlsanity.com/paper/2210.03709/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 102 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved](https://mlsanity.com/paper/1811.11154/) (2018)
- [Discrimination in Online Ad Delivery](https://mlsanity.com/paper/1301.6822/) (2013)
- [Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance](https://mlsanity.com/paper/2206.04737/) (2022)
- [Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support](https://mlsanity.com/paper/2112.05675/) (2021)
- [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://mlsanity.com/paper/1904.01121/) (2019)
- [Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors](https://mlsanity.com/paper/2105.02980/) (2021)
- [Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure](https://mlsanity.com/paper/1809.07424/) (2018)
- [Roles for Computing in Social Change](https://mlsanity.com/paper/1912.04883/) (2019)
- [Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices](https://mlsanity.com/paper/2006.12358/) (2020)
- [Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits](https://mlsanity.com/paper/2205.06922/) (2022)
- [Discovering and Validating AI Errors With Crowdsourced Failure Reports](https://mlsanity.com/paper/2109.11690/) (2021)
- [What If I Don't Like Any Of The Choices? The Limits of Preference Elicitation for Participatory Algorithm Design](https://mlsanity.com/paper/2007.06718/) (2020)
- [Empowering Local Communities Using Artificial Intelligence](https://mlsanity.com/paper/2110.02007/) (2021)

## Built on by (most cited first, 30 of 32)

- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia](https://mlsanity.com/paper/2402.14147/) (2024)
- [Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"](https://mlsanity.com/paper/2302.01448/) (2023)
- [Investigating Practices and Opportunities for Cross-functional Collaboration around AI Fairness in Industry Practice](https://mlsanity.com/paper/2306.06542/) (2023)
- [AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development](https://mlsanity.com/paper/2502.18682/) (2025)
- [Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMs](https://mlsanity.com/paper/2310.09668/) (2023)
- [PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming](https://mlsanity.com/paper/2509.03728/) (2025)
- [Clarify: Improving Model Robustness With Natural Language Corrections](https://mlsanity.com/paper/2402.03715/) (2024)
- [Understanding Practices around Computational News Discovery Tools in the Domain of Science Journalism](https://mlsanity.com/paper/2311.06864/) (2023)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [A Systematic Literature Review of Human-Centered, Ethical, and Responsible AI](https://mlsanity.com/paper/2302.05284/) (2023)
- [Robust ML Auditing using Prior Knowledge](https://mlsanity.com/paper/2505.04796/) (2025)
- [Under manipulations, are some AI models harder to audit?](https://mlsanity.com/paper/2402.09043/) (2024)
- ["It Might be Technically Impressive, But It's Practically Useless to us": Motivations, Practices, Challenges, and Opportunities for Cross-Functional Collaboration around AI within the News Industry](https://mlsanity.com/paper/2409.12000/) (2024)
- [Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors](https://mlsanity.com/paper/2409.18203/) (2024)
- [MIRAGE: Multi-model Interface for Reviewing and Auditing Generative Text-to-Image AI](https://mlsanity.com/paper/2503.19252/) (2025)
- [Attention to Non-Adopters](https://mlsanity.com/paper/2510.15951/) (2025)
- [The Value-Sensitive Conversational Agent Co-Design Framework](https://mlsanity.com/paper/2310.11848/) (2023)
- [(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court](https://mlsanity.com/paper/2403.13004/) (2024)
- [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [AI LEGO: Scaffolding Cross-Functional Collaboration in Industrial Responsible AI Practices during Early Design Stages](https://mlsanity.com/paper/2505.10300/) (2025)
- [From Reflection to Repair: A Scoping Review of Dataset Documentation Tools](https://mlsanity.com/paper/2602.15968/) (2026)
- [Alignment Is Not Enough: A Relational Framework for Moral Standing in Human-AI Interaction](https://mlsanity.com/paper/2603.00078/) (2026)
- [Human-Centred LLM Privacy Audits: Findings and Frictions](https://mlsanity.com/paper/2603.12094/) (2026)
- [Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems](https://mlsanity.com/paper/2505.19441/) (2025)
- [From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models](https://mlsanity.com/paper/2507.17922/) (2025)
- [Seeing Twice: How Side-by-Side T2I Comparison Changes Auditing Strategies](https://mlsanity.com/paper/2511.21547/) (2025)
- [Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma](https://mlsanity.com/paper/2603.13294/) (2026)
- [Bureaucratic Silences: What the Canadian AI Register Reveals, Omits, and Obscures](https://mlsanity.com/paper/2604.15514/) (2026)

## Abstract (arXiv)

Recent years have seen growing interest among both researchers and practitioners in user-engaged approaches to algorithm auditing, which directly engage users in detecting problematic behaviors in algorithmic systems. However, we know little about industry practitioners' current practices and challenges around user-engaged auditing, nor what opportunities exist for them to better leverage such approaches in practice. To investigate, we conducted a series of interviews and iterative co-design activities with practitioners who employ user-engaged auditing approaches in their work. Our findings reveal several challenges practitioners face in appropriately recruiting and incentivizing user auditors, scaffolding user audits, and deriving actionable insights from user-engaged audit reports. Furthermore, practitioners shared organizational obstacles to user-engaged auditing, surfacing a complex relationship between practitioners and user auditors. Based on these findings, we discuss opportunities for future HCI research to help realize the potential (and the mitigate risks) of user-engaged auditing in industry practice.