# JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models

arXiv: https://arxiv.org/abs/2406.15484 · HTML: https://mlsanity.com/paper/2406.15484/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations](https://mlsanity.com/paper/2308.02053/) (2023)
- [Quantifying Social Biases Using Templates is Unreliable](https://mlsanity.com/paper/2210.04337/) (2022)
- [Application of LLM Agents in Recruitment: A Novel Framework for Resume Screening](https://mlsanity.com/paper/2401.08315/) (2024)
- [Auditing the Use of Language Models to Guide Hiring Decisions](https://mlsanity.com/paper/2404.03086/) (2024)
- [Hire Me or Not? Examining Language Model's Behavior with Occupation Attributes](https://mlsanity.com/paper/2405.06687/) (2024)

## Built on by (most cited first, 21 of 21)

- [Robustly Improving LLM Fairness in Realistic Settings via Interpretability](https://mlsanity.com/paper/2506.10922/) (2025)
- [Bias Amplification: Large Language Models as Increasingly Biased Media](https://mlsanity.com/paper/2410.15234/) (2024)
- [Natural Language Processing for Human Resources: A Survey](https://mlsanity.com/paper/2410.16498/) (2024)
- [Evaluating the Promise and Pitfalls of LLMs in Hiring Decisions](https://mlsanity.com/paper/2507.02087/) (2025)
- [Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models](https://mlsanity.com/paper/2506.14092/) (2025)
- [On the Mutual Influence of Gender and Occupation in LLM Representations](https://mlsanity.com/paper/2503.06792/) (2025)
- [Different Bias Under Different Criteria: Assessing Bias in LLMs with a Fact-Based Approach](https://mlsanity.com/paper/2411.17338/) (2024)
- [Emergence of Biased Consensus in Multi-Agent LLM Debates](https://mlsanity.com/paper/2608.02827/) (2026)
- [MPF: Aligning and Debiasing Language Models post Deployment via Multi Perspective Fusion](https://mlsanity.com/paper/2507.02595/) (2025)
- [Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System](https://mlsanity.com/paper/2602.14970/) (2026)
- [Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers](https://mlsanity.com/paper/2606.02776/) (2026)
- [Identifying and Mitigating Gender Cues in Academic Recommendation Letters: An Interpretability Case Study](https://mlsanity.com/paper/2604.12337/) (2026)
- [BiasLab: A Multilingual Dual-Framing Framework for LLM Bias Measurement, Applied to Workplace and HR Contexts](https://mlsanity.com/paper/2601.06861/) (2026)
- [Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents](https://mlsanity.com/paper/2604.18122/) (2026)
- [Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit](https://mlsanity.com/paper/2605.30804/) (2026)
- [The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data](https://mlsanity.com/paper/2608.04268/) (2026)
- [Status Association Does Not Reliably Predict Decision Leakage](https://mlsanity.com/paper/2608.10089/) (2026)
- [No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation](https://mlsanity.com/paper/2608.21206/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable](https://mlsanity.com/paper/2609.04127/) (2026)
- [Monocultural Biases: Correlated biases in large language models lead to unequal systemic exclusion rates in hiring](https://mlsanity.com/paper/2609.22169/) (2026)

## Abstract (arXiv)

The use of Large Language Models (LLMs) in hiring has led to legislative actions to protect vulnerable demographic groups. This paper presents a novel framework for benchmarking hierarchical gender hiring bias in Large Language Models (LLMs) for resume scoring, revealing significant issues of reverse gender hiring bias and overdebiasing. Our contributions are fourfold: Firstly, we introduce a new construct grounded in labour economics, legal principles, and critiques of current bias benchmarks: hiring bias can be categorized into two types: Level bias (difference in the average outcomes between demographic counterfactual groups) and Spread bias (difference in the variance of outcomes between demographic counterfactual groups); Level bias can be further subdivided into statistical bias (i.e. changing with non-demographic content) and taste-based bias (i.e. consistent regardless of non-demographic content). Secondly, the framework includes rigorous statistical and computational hiring bias metrics, such as Rank After Scoring (RAS), Rank-based Impact Ratio, Permutation Test, and Fixed Effects Model. Thirdly, we analyze gender hiring biases in ten state-of-the-art LLMs. Seven out of ten LLMs show significant biases against males in at least one industry. An industry-effect regression reveals that the healthcare industry is the most biased against males. Moreover, we found that the bias performance remains invariant with resume content for eight out of ten LLMs. This indicates that the bias performance measured in this paper might apply to other resume datasets with different resume qualities. Fourthly, we provide a user-friendly demo and resume dataset to support the adoption and practical use of the framework, which can be generalized to other social traits and tasks.