# The Birth of Bias: A case study on the evolution of gender bias in an English language model arXiv: https://arxiv.org/abs/2207.10245 ยท HTML: https://mlsanity.com/paper/2207.10245/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 49 references that are in the map. - First posted on arXiv 21 Jul 2022. ## Builds on (most cited first, 30 of 35) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016) - [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019) - [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021) - [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020) - [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016) - [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018) - [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018) - [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020) - [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020) - [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018) - [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019) - [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019) - [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016) - [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://mlsanity.com/paper/2010.06032/) (2020) - [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018) - [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018) - [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019) - [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020) - [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018) - [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020) - [Assessing Social and Intersectional Biases in Contextualized Word Representations](https://mlsanity.com/paper/1911.01485/) (2019) - [It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution](https://mlsanity.com/paper/1909.00871/) (2019) - [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019) - [Incorporating Priors with Feature Attribution on Text Classification](https://mlsanity.com/paper/1906.08286/) (2019) - [Understanding Undesirable Word Embedding Associations](https://mlsanity.com/paper/1908.06361/) (2019) - [Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation](https://mlsanity.com/paper/2005.00965/) (2020) - [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021) - [Reliability Testing for Natural Language Processing Systems](https://mlsanity.com/paper/2105.02590/) (2021) ## Built on by (most cited first, 16 of 16) - [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023) - [PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs](https://mlsanity.com/paper/2503.09543/) (2025) - [Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications](https://mlsanity.com/paper/2307.09162/) (2023) - [Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model](https://mlsanity.com/paper/2310.12611/) (2023) - [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024) - [Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation](https://mlsanity.com/paper/2505.08546/) (2025) - [Hire Me or Not? Examining Language Model's Behavior with Occupation Attributes](https://mlsanity.com/paper/2405.06687/) (2024) - [Undesirable Biases in NLP: Addressing Challenges of Measurement](https://mlsanity.com/paper/2211.13709/) (2022) - [The Cultivated Practices of Text-to-Image Generation](https://mlsanity.com/paper/2306.11393/) (2023) - [Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Models](https://mlsanity.com/paper/2504.03029/) (2025) - [Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale](https://mlsanity.com/paper/2306.05036/) (2023) - [Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translation](https://mlsanity.com/paper/2501.10150/) (2025) - [Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures](https://mlsanity.com/paper/2603.17952/) (2026) - [Black Big Boxes: Tracing Adjective Order Preferences in Large Language Models](https://mlsanity.com/paper/2407.02136/) (2024) - [Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior](https://mlsanity.com/paper/2510.14261/) (2025) - [Is She Even Relevant? When BERT Ignores Explicit Gender Cues](https://mlsanity.com/paper/2605.07622/) (2026) ## Abstract (arXiv) Detecting and mitigating harmful biases in modern language models are widely recognized as crucial, open problems. In this paper, we take a step back and investigate how language models come to be biased in the first place. We use a relatively small language model, using the LSTM architecture trained on an English Wikipedia corpus. With full access to the data and to the model parameters as they change during every step while training, we can map in detail how the representation of gender develops, what patterns in the dataset drive this, and how the model's internal state relates to the bias in a downstream task (semantic textual similarity). We find that the representation of gender is dynamic and identify different phases during training. Furthermore, we show that gender information is represented increasingly locally in the input embeddings of the model and that, as a consequence, debiasing these can be effective in reducing the downstream bias. Monitoring the training dynamics, allows us to detect an asymmetry in how the female and male gender are represented in the input embeddings. This is important, as it may cause naive mitigation strategies to introduce new undesirable biases. We discuss the relevance of the findings for mitigation strategies more generally and the prospects of generalizing our methods to larger language models, the Transformer architecture, other languages and other undesirable biases.