# Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising

arXiv: https://arxiv.org/abs/2308.15768 · HTML: https://mlsanity.com/paper/2308.15768/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2023.

## Built on by (most cited first, 12 of 12)

- [Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia](https://mlsanity.com/paper/2402.14147/) (2024)
- [From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing](https://mlsanity.com/paper/2410.04772/) (2024)
- [Robust ML Auditing using Prior Knowledge](https://mlsanity.com/paper/2505.04796/) (2025)
- [Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers](https://mlsanity.com/paper/2410.22937/) (2024)
- [HCC Is All You Need: Alignment-The Sensible Kind Anyway-Is Just Human-Centered Computing](https://mlsanity.com/paper/2405.03699/) (2024)
- [Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives](https://mlsanity.com/paper/2509.05627/) (2025)
- [Why am I Still Seeing This: Measuring the Effectiveness Of Ad Controls and Explanations in AI-Mediated Ad Targeting Systems](https://mlsanity.com/paper/2408.11910/) (2024)
- [FairFare: A Tool for Crowdsourcing Rideshare Data to Empower Labor Organizers](https://mlsanity.com/paper/2502.11273/) (2025)
- [Learning About Algorithm Auditing in Five Steps: Scaffolding How High School Youth Can Systematically and Critically Evaluate Machine Learning Applications](https://mlsanity.com/paper/2412.06989/) (2024)
- [Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation](https://mlsanity.com/paper/2602.18483/) (2026)
- [Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software](https://mlsanity.com/paper/2604.10875/) (2026)
- [Attribute Inference from Interactive Targeted Ads](https://mlsanity.com/paper/2606.15209/) (2026)

## Abstract (arXiv)

Algorithm audits are powerful tools for studying black-box systems. While very effective in examining technical components, the method stops short of a sociotechnical frame, which would also consider users as an integral and dynamic part of the system. Addressing this gap, we propose the concept of sociotechnical auditing: auditing methods that evaluate algorithmic systems at the sociotechnical level, focusing on the interplay between algorithms and users as each impacts the other. Just as algorithm audits probe an algorithm with varied inputs and observe outputs, a sociotechnical audit (STA) additionally probes users, exposing them to different algorithmic behavior and measuring resulting attitudes and behaviors. To instantiate this method, we develop Intervenr, a platform for conducting browser-based, longitudinal sociotechnical audits with consenting, compensated participants. Intervenr investigates the algorithmic content users encounter online and coordinates systematic client-side interventions to understand how users change in response. As a case study, we deploy Intervenr in a two-week sociotechnical audit of online advertising (N=244) to investigate the central premise that personalized ad targeting is more effective on users. In the first week, we collect all browser ads delivered to users, and in the second, we deploy an ablation-style intervention that disrupts normal targeting by randomly pairing participants and swapping all their ads. We collect user-oriented metrics (self-reported ad interest and feeling of representation) and advertiser-oriented metrics (ad views, clicks, and recognition) throughout, along with a total of over 500,000 ads. Our STA finds that targeted ads indeed perform better with users, but also that users begin to acclimate to different ads in only a week, casting doubt on the primacy of personalized ad targeting given the impact of repeated exposure.