# Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2110.05367 · HTML: https://mlsanity.com/paper/2110.05367/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://mlsanity.com/paper/2010.06032/) (2020)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Assessing Social and Intersectional Biases in Contextualized Word Representations](https://mlsanity.com/paper/1911.01485/) (2019)
- [FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders](https://mlsanity.com/paper/2103.06413/) (2021)
- [Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function](https://mlsanity.com/paper/1905.12801/) (2019)
- [Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models](https://mlsanity.com/paper/2101.09688/) (2021)
- [They, Them, Theirs: Rewriting with Gender-Neutral English](https://mlsanity.com/paper/2102.06788/) (2021)
- [WikiCREM: A Large Unsupervised Corpus for Coreference Resolution](https://mlsanity.com/paper/1908.08025/) (2019)

## Built on by (most cited first, 30 of 35)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes](https://mlsanity.com/paper/2402.01981/) (2024)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases](https://mlsanity.com/paper/2112.07868/) (2021)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [From 'Showgirls' to 'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs](https://mlsanity.com/paper/2407.04434/) (2024)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Potential and Challenges of Model Editing for Social Debiasing](https://mlsanity.com/paper/2402.13462/) (2024)
- [The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation](https://mlsanity.com/paper/2501.07849/) (2025)
- [Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Bias](https://mlsanity.com/paper/2204.10365/) (2022)
- [Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions](https://mlsanity.com/paper/2211.11798/) (2022)
- [Fairness in Large Language Models in Three Hours](https://mlsanity.com/paper/2408.00992/) (2024)
- [Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance](https://mlsanity.com/paper/2512.23461/) (2025)
- [Improving LLM Group Fairness on Tabular Data via In-Context Learning](https://mlsanity.com/paper/2412.04642/) (2024)
- [More of the Same: Persistent Representational Harms Under Increased Representation](https://mlsanity.com/paper/2503.00333/) (2025)
- [Exposing Product Bias in LLM Investment Recommendation](https://mlsanity.com/paper/2503.08750/) (2025)
- [Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation](https://mlsanity.com/paper/2412.14050/) (2024)
- [Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms](https://mlsanity.com/paper/2503.04372/) (2025)
- [Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs](https://mlsanity.com/paper/2507.07186/) (2025)
- [LIDAO: Towards Limited Interventions for Debiasing (Large) Language Models](https://mlsanity.com/paper/2406.00548/) (2024)
- [Specification Overfitting in Artificial Intelligence](https://mlsanity.com/paper/2403.08425/) (2024)
- [Expert-Guided Extinction of Toxic Tokens for Debiased Generation](https://mlsanity.com/paper/2405.19299/) (2024)
- [Collapsed Language Models Promote Fairness](https://mlsanity.com/paper/2410.04472/) (2024)
- [Learning Horn Envelopes via Queries from Large Language Models](https://mlsanity.com/paper/2305.12143/) (2023)

## Abstract (arXiv)

Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training on the model with such data. However, given the limited size and concentrated focus of the gender-neutral data, catastrophic forgetting would occur during second-phase pre-training. Forgetting information in the original training data may damage the model's downstream performance by a large margin. In this work, we empirically show that catastrophic forgetting occurs in such methods by evaluating them with general NLP tasks in GLUE. Then, we propose a new method, GEnder Equality Prompt (GEEP), to improve gender fairness of pre-trained models with less forgetting. GEEP freezes the pre-trained model and learns gender-related prompts with gender-neutral data. Empirical results show that GEEP not only achieves SOTA performances on gender fairness tasks, but also forgets less and performs better on GLUE by a large margin.