# Auditing for Discrimination in Algorithms Delivering Job Ads

arXiv: https://arxiv.org/abs/2104.04502 · HTML: https://mlsanity.com/paper/2104.04502/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2021.

## Built on by (most cited first, 17 of 17)

- [Evaluating and Mitigating Discrimination in Language Model Decisions](https://mlsanity.com/paper/2312.03689/) (2023)
- [Improved Approximation Algorithms for Individually Fair Clustering](https://mlsanity.com/paper/2106.14043/) (2021)
- [Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2209.11812/) (2022)
- [Online certification of preference-based fairness for personalized recommender systems](https://mlsanity.com/paper/2104.14527/) (2021)
- ["There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making](https://mlsanity.com/paper/2205.05758/) (2022)
- [A Survey on Fairness-aware Recommender Systems](https://mlsanity.com/paper/2306.00403/) (2023)
- [Fast online ranking with fairness of exposure](https://mlsanity.com/paper/2209.13019/) (2022)
- [Towards Trustworthy AI-Empowered Real-Time Bidding for Online Advertisement Auctioning](https://mlsanity.com/paper/2210.07770/) (2022)
- [Perceptions of Fairness and Trustworthiness Based on Explanations in Human vs. Automated Decision-Making](https://mlsanity.com/paper/2109.05792/) (2021)
- [Unmasking Gender Bias in Recommendation Systems and Enhancing Category-Aware Fairness](https://mlsanity.com/paper/2502.17921/) (2025)
- [A Human-Centric Perspective on Fairness and Transparency in Algorithmic Decision-Making](https://mlsanity.com/paper/2205.00033/) (2022)
- [Why am I Still Seeing This: Measuring the Effectiveness Of Ad Controls and Explanations in AI-Mediated Ad Targeting Systems](https://mlsanity.com/paper/2408.11910/) (2024)
- [When Ads Become Profiles: Uncovering the Invisible Risk of Web Advertising at Scale with LLMs](https://mlsanity.com/paper/2509.18874/) (2025)
- [Automatic Authorities: Power and AI](https://mlsanity.com/paper/2404.05990/) (2024)
- [Perceptions of Discriminatory Decisions of Artificial Intelligence: Unpacking the Role of Individual Characteristics](https://mlsanity.com/paper/2410.13250/) (2024)
- [Privacy-Friendly Cohort Determination: Sealed, CSP-Independent In-Browser ML Inference of Professional Segments for Identity-Less Advertising](https://mlsanity.com/paper/2609.36153/) (2026)
- [AI Representations of Mental Health Stigma: Quantifying the Gap Between AI Estimations and Human Perceptions Across Gender Groups Using GPT-Based LLMs](https://mlsanity.com/paper/2610.16042/) (2026)

## Abstract (arXiv)

Ad platforms such as Facebook, Google and LinkedIn promise value for advertisers through their targeted advertising. However, multiple studies have shown that ad delivery on such platforms can be skewed by gender or race due to hidden algorithmic optimization by the platforms, even when not requested by the advertisers. Building on prior work measuring skew in ad delivery, we develop a new methodology for black-box auditing of algorithms for discrimination in the delivery of job advertisements. Our first contribution is to identify the distinction between skew in ad delivery due to protected categories such as gender or race, from skew due to differences in qualification among people in the targeted audience. This distinction is important in U.S. law, where ads may be targeted based on qualifications, but not on protected categories. Second, we develop an auditing methodology that distinguishes between skew explainable by differences in qualifications from other factors, such as the ad platform's optimization for engagement or training its algorithms on biased data. Our method controls for job qualification by comparing ad delivery of two concurrent ads for similar jobs, but for a pair of companies with different de facto gender distributions of employees. We describe the careful statistical tests that establish evidence of non-qualification skew in the results. Third, we apply our proposed methodology to two prominent targeted advertising platforms for job ads: Facebook and LinkedIn. We confirm skew by gender in ad delivery on Facebook, and show that it cannot be justified by differences in qualifications. We fail to find skew in ad delivery on LinkedIn. Finally, we suggest improvements to ad platform practices that could make external auditing of their algorithms in the public interest more feasible and accurate.