# Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting

arXiv: https://arxiv.org/abs/1901.09451 · HTML: https://mlsanity.com/paper/1901.09451/

## Key facts

- Cited by 349 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2019.

## Built on by (most cited first, 30 of 349)

- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations](https://mlsanity.com/paper/2203.13928/) (2022)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Post-processing for Individual Fairness](https://mlsanity.com/paper/2110.13796/) (2021)

## Abstract (arXiv)

We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators---such as first names and pronouns---in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are "scrubbed," and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.