# Auditing the Use of Language Models to Guide Hiring Decisions

arXiv: https://arxiv.org/abs/2404.03086 · HTML: https://mlsanity.com/paper/2404.03086/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 61 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Avoiding Discrimination through Causal Reasoning](https://mlsanity.com/paper/1706.02744/) (2017)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [Fair Inference On Outcomes](https://mlsanity.com/paper/1705.10378/) (2017)
- [Evaluating and Mitigating Discrimination in Language Model Decisions](https://mlsanity.com/paper/2312.03689/) (2023)
- [The Measure and Mismeasure of Fairness](https://mlsanity.com/paper/1808.00023/) (2018)
- [Counterfactual Risk Assessments, Evaluation, and Fairness](https://mlsanity.com/paper/1909.00066/) (2019)
- [Causal Reasoning for Algorithmic Fairness](https://mlsanity.com/paper/1805.05859/) (2018)
- [What's in a Name? Auditing Large Language Models for Race and Gender Bias](https://mlsanity.com/paper/2402.14875/) (2024)
- [Are Emily and Greg Still More Employable than Lakisha and Jamal? Investigating Algorithmic Hiring Bias in the Era of ChatGPT](https://mlsanity.com/paper/2310.05135/) (2023)
- [Principal Fairness for Human and Algorithmic Decision-Making](https://mlsanity.com/paper/2005.10400/) (2020)
- [Auditing Work: Exploring the New York City algorithmic bias audit regime](https://mlsanity.com/paper/2402.08101/) (2024)
- [Equal Opportunity and Affirmative Action via Counterfactual Predictions](https://mlsanity.com/paper/1905.10870/) (2019)
- [A Causal Framework for Observational Studies of Discrimination](https://mlsanity.com/paper/2006.12460/) (2020)

## Built on by (most cited first, 20 of 20)

- [Correlated Errors in Large Language Models](https://mlsanity.com/paper/2506.07962/) (2025)
- [Evaluating language models as risk scores](https://mlsanity.com/paper/2407.14614/) (2024)
- [Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?](https://mlsanity.com/paper/2406.10486/) (2024)
- [JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models](https://mlsanity.com/paper/2406.15484/) (2024)
- [Evaluating Bias in LLMs for Job-Resume Matching: Gender, Race, and Education](https://mlsanity.com/paper/2503.19182/) (2025)
- [Evaluating the Promise and Pitfalls of LLMs in Hiring Decisions](https://mlsanity.com/paper/2507.02087/) (2025)
- [Unboxing Occupational Bias: Grounded Debiasing of LLMs with U.S. Labor Data](https://mlsanity.com/paper/2408.11247/) (2024)
- [Towards Effective Discrimination Testing for Generative AI](https://mlsanity.com/paper/2412.21052/) (2024)
- [Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models](https://mlsanity.com/paper/2506.14092/) (2025)
- [Do Prevalent Bias Metrics Capture Allocational Harms from LLMs?](https://mlsanity.com/paper/2408.01285/) (2024)
- [Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks](https://mlsanity.com/paper/2603.05801/) (2026)
- [Small Changes, Large Consequences: Analyzing the Allocational Fairness of LLMs in Hiring Contexts](https://mlsanity.com/paper/2501.04316/) (2025)
- [Fairness Is Not Enough: Auditing Competence and Intersectional Bias in AI-powered Resume Screening](https://mlsanity.com/paper/2507.11548/) (2025)
- ["You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations](https://mlsanity.com/paper/2510.19167/) (2025)
- [Evaluating LLM Behavior in Hiring: Implicit Weights, Fairness Across Groups, and Alignment with Human Preferences](https://mlsanity.com/paper/2601.11379/) (2026)
- [Pro-AI Bias in Large Language Models](https://mlsanity.com/paper/2601.13749/) (2026)
- [LLM Nepotism in Organizational Governance](https://mlsanity.com/paper/2604.09620/) (2026)
- [How Supply Chain Dependencies Complicate Bias Measurement and Accountability Attribution in AI Hiring Applications](https://mlsanity.com/paper/2604.22679/) (2026)
- [Operationalizing Individual Fairness via Gradient Descent and Bradley-Terry Models](https://mlsanity.com/paper/2605.23145/) (2026)
- [Fairness Beyond Anonymization? Demographic Leakage in German LLM-Generated Resumes](https://mlsanity.com/paper/2609.22188/) (2026)

## Abstract (arXiv)

Regulatory efforts to protect against algorithmic bias have taken on increased urgency with rapid advances in large language models (LLMs), which are machine learning models that can achieve performance rivaling human experts on a wide array of tasks. A key theme of these initiatives is algorithmic "auditing," but current regulations -- as well as the scientific literature -- provide little guidance on how to conduct these assessments. Here we propose and investigate one approach for auditing algorithms: correspondence experiments, a widely applied tool for detecting bias in human judgements. In the employment context, correspondence experiments aim to measure the extent to which race and gender impact decisions by experimentally manipulating elements of submitted application materials that suggest an applicant's demographic traits, such as their listed name. We apply this method to audit candidate assessments produced by several state-of-the-art LLMs, using a novel corpus of applications to K-12 teaching positions in a large public school district. We find evidence of moderate race and gender disparities, a pattern largely robust to varying the types of application material input to the models, as well as the framing of the task to the LLMs. We conclude by discussing some important limitations of correspondence experiments for auditing algorithms.