# The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations

arXiv: https://arxiv.org/abs/2308.02053 · HTML: https://mlsanity.com/paper/2308.02053/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models](https://mlsanity.com/paper/2304.03738/) (2023)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [The Self-Perception and Political Biases of ChatGPT](https://mlsanity.com/paper/2304.07333/) (2023)
- [Quantifying Social Biases Using Templates is Unreliable](https://mlsanity.com/paper/2210.04337/) (2022)
- [On Natural Language User Profiles for Transparent and Scrutable Recommendation](https://mlsanity.com/paper/2205.09403/) (2022)

## Built on by (most cited first, 30 of 41)

- [Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment](https://mlsanity.com/paper/2501.09620/) (2025)
- [Stereotype or Personalization? User Identity Biases Chatbot Recommendations](https://mlsanity.com/paper/2410.05613/) (2024)
- ["You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations](https://mlsanity.com/paper/2406.12232/) (2024)
- [JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models](https://mlsanity.com/paper/2406.15484/) (2024)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards](https://mlsanity.com/paper/2403.13213/) (2024)
- ["Global is Good, Local is Bad?": Understanding Brand Bias in LLMs](https://mlsanity.com/paper/2406.13997/) (2024)
- [From 'Showgirls' to 'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs](https://mlsanity.com/paper/2407.04434/) (2024)
- [Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs](https://mlsanity.com/paper/2406.13993/) (2024)
- [Gender Bias in LLM-generated Interview Responses](https://mlsanity.com/paper/2410.20739/) (2024)
- [DAIQ: Auditing Demographic Attribute Inference from Question in LLMs](https://mlsanity.com/paper/2508.15830/) (2025)
- [Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases](https://mlsanity.com/paper/2509.04373/) (2025)
- [Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models](https://mlsanity.com/paper/2504.05325/) (2025)
- [Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs](https://mlsanity.com/paper/2505.02456/) (2025)
- [Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification](https://mlsanity.com/paper/2505.19776/) (2025)
- [Data Feminism for AI](https://mlsanity.com/paper/2405.01286/) (2024)
- [Evaluating how LLM annotations represent diverse views on contentious topics](https://mlsanity.com/paper/2503.23243/) (2025)
- [How far can bias go? Tracing bias from pretraining data to alignment](https://mlsanity.com/paper/2411.19240/) (2024)
- [More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models](https://mlsanity.com/paper/2503.15904/) (2025)
- [Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectives](https://mlsanity.com/paper/2407.18376/) (2024)
- [Gender and Positional Biases in LLM-Based Hiring Decisions: Evidence from Comparative CV/R'esum'e Evaluations](https://mlsanity.com/paper/2505.17049/) (2025)
- [Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution](https://mlsanity.com/paper/2508.07111/) (2025)
- [AI in Mental Health: Emotional and Sentiment Analysis of Large Language Models' Responses to Depression, Anxiety, and Stress Queries](https://mlsanity.com/paper/2508.11285/) (2025)
- [Whose Journey Matters? Investigating Identity Biases in Large Language Models (LLMs) for Travel Planning Assistance](https://mlsanity.com/paper/2410.17333/) (2024)
- [Revisiting Rogers' Paradox in the Context of Human-AI Interaction](https://mlsanity.com/paper/2501.10476/) (2025)
- [Breaking Down Bias: On The Limits of Generalizable Pruning Strategies](https://mlsanity.com/paper/2502.07771/) (2025)
- [Do LLMs have a Gender (Entropy) Bias?](https://mlsanity.com/paper/2505.20343/) (2025)
- [Information-Consistent Language Model Recommendations through Group Relative Policy Optimization](https://mlsanity.com/paper/2512.12858/) (2025)
- [Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos](https://mlsanity.com/paper/2601.06931/) (2026)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have seen widespread deployment in various real-world applications. Understanding these biases is crucial to comprehend the potential downstream consequences when using LLMs to make decisions, particularly for historically disadvantaged groups. In this work, we propose a simple method for analyzing and comparing demographic bias in LLMs, through the lens of job recommendations. We demonstrate the effectiveness of our method by measuring intersectional biases within ChatGPT and LLaMA, two cutting-edge LLMs. Our experiments primarily focus on uncovering gender identity and nationality bias; however, our method can be extended to examine biases associated with any intersection of demographic identities. We identify distinct biases in both models toward various demographic identities, such as both models consistently suggesting low-paying jobs for Mexican workers or preferring to recommend secretarial roles to women. Our study highlights the importance of measuring the bias of LLMs in downstream applications to understand the potential for harm and inequitable outcomes.