mlsanityData to 14 Oct 2026

JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models

arXiv 2406.15484 ↗First posted 17 Jun 2024cs.CL cs.AI cs.CYcited by 21

Not on the map. Why?

Abstract (arXiv)

The use of Large Language Models (LLMs) in hiring has led to legislative actions to protect vulnerable demographic groups. This paper presents a novel framework for benchmarking hierarchical gender hiring bias in Large Language Models (LLMs) for resume scoring, revealing significant issues of reverse gender hiring bias and overdebiasing. Our contributions are fourfold: Firstly, we introduce a new construct grounded in labour economics, legal principles, and critiques of current bias benchmarks: hiring bias can be categorized into two types: Level bias (difference in the average outcomes between demographic counterfactual groups) and Spread bias (difference in the variance of outcomes between demographic counterfactual groups); Level bias can be further subdivided into statistical bias (i.e. changing with non-demographic content) and taste-based bias (i.e. consistent regardless of non-demographic content). Secondly, the framework includes rigorous statistical and computational hiring bias metrics, such as Rank After Scoring (RAS), Rank-based Impact Ratio, Permutation Test, and Fixed Effects Model. Thirdly, we analyze gender hiring biases in ten state-of-the-art LLMs. Seven out of ten LLMs show significant biases against males in at least one industry. An industry-effect regression reveals that the healthcare industry is the most biased against males. Moreover, we found that the bias performance remains invariant with resume content for eight out of ten LLMs. This indicates that the bias performance measured in this paper might apply to other resume datasets with different resume qualities. Fourthly, we provide a user-friendly demo and resume dataset to support the adoption and practical use of the framework, which can be generalized to other social traits and tasks.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 11 of its 34 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

11 of 34 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2017Counterfactual Fairnesscited by 804
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  7. 2023The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendationscited by 41
  8. 2022Quantifying Social Biases Using Templates is Unreliablecited by 39
  9. 2024Application of LLM Agents in Recruitment: A Novel Framework for Resume Screeningcited by 24
  10. 2024Auditing the Use of Language Models to Guide Hiring Decisionscited by 20
  11. 2024Hire Me or Not? Examining Language Model's Behavior with Occupation Attributescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025Robustly Improving LLM Fairness in Realistic Settings via Interpretabilitycited by 18
  2. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  3. 2024Natural Language Processing for Human Resources: A Surveycited by 10
  4. 2025Evaluating the Promise and Pitfalls of LLMs in Hiring Decisionscited by 7
  5. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 5
  6. 2025On the Mutual Influence of Gender and Occupation in LLM Representationscited by 4
  7. 2024Different Bias Under Different Criteria: Assessing Bias in LLMs with a Fact-Based Approachcited by 3
  8. 2026Emergence of Biased Consensus in Multi-Agent LLM Debatescited by 3
  9. 2025MPF: Aligning and Debiasing Language Models post Deployment via Multi Perspective Fusioncited by 2
  10. 2026Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance Systemcited by 2
  11. 2026Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answerscited by 2
  12. 2026Identifying and Mitigating Gender Cues in Academic Recommendation Letters: An Interpretability Case Studycited by 1
9 more
  1. 2026BiasLab: A Multilingual Dual-Framing Framework for LLM Bias Measurement, Applied to Workplace and HR Contextscited by 0
  2. 2026Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documentscited by 0
  3. 2026Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Auditcited by 0
  4. 2026The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Datacited by 0
  5. 2026Status Association Does Not Reliably Predict Decision Leakagecited by 0
  6. 2026No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluationcited by 0
  7. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  8. 2026Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailablecited by 0
  9. 2026Monocultural Biases: Correlated biases in large language models lead to unequal systemic exclusion rates in hiringcited by 0