# Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning

arXiv: https://arxiv.org/abs/2207.02463 · HTML: https://mlsanity.com/paper/2207.02463/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 6 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Gender-preserving Debiasing for Pre-trained Word Embeddings](https://mlsanity.com/paper/1906.00742/) (2019)
- [PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction](https://mlsanity.com/paper/2010.13816/) (2020)
- [Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets](https://mlsanity.com/paper/1911.09241/) (2019)

## Built on by (most cited first, 21 of 21)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [A Trip Towards Fairness: Bias and De-Biasing in Large Language Models](https://mlsanity.com/paper/2305.13862/) (2023)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [BiasEdit: Debiasing Stereotyped Language Models via Model Editing](https://mlsanity.com/paper/2503.08588/) (2025)
- [In-Contextual Gender Bias Suppression for Large Language Models](https://mlsanity.com/paper/2309.07251/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [A Multi-LLM Debiasing Framework](https://mlsanity.com/paper/2409.13884/) (2024)
- [Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings](https://mlsanity.com/paper/2412.06134/) (2024)
- [Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms](https://mlsanity.com/paper/2503.04372/) (2025)
- [Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Models](https://mlsanity.com/paper/2503.06011/) (2025)
- [Using Semantic Similarity and Text Embedding to Measure the Social Media Echo of Strategic Communications](https://mlsanity.com/paper/2303.16694/) (2023)
- [Unlocking Bias Detection: Leveraging Transformer-Based Models for Content Analysis](https://mlsanity.com/paper/2310.00347/) (2023)
- [GIEBench: Towards Holistic Evaluation of Group Identity-based Empathy for Large Language Models](https://mlsanity.com/paper/2406.14903/) (2024)
- [Local Contrastive Editing of Gender Stereotypes](https://mlsanity.com/paper/2410.17739/) (2024)
- [Gender Bias in Transformer Models: A comprehensive survey](https://mlsanity.com/paper/2306.10530/) (2023)
- [LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models](https://mlsanity.com/paper/2505.15475/) (2025)
- [Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment](https://mlsanity.com/paper/2412.03400/) (2024)
- [A Multifaceted Analysis of Social Biases in Large Language Models](https://mlsanity.com/paper/2512.15792/) (2025)
- [A Representation-Level Assessment of Bias Mitigation in Foundation Models](https://mlsanity.com/paper/2604.08561/) (2026)

## Abstract (arXiv)

Language model debiasing has emerged as an important field of study in the NLP community. Numerous debiasing techniques were proposed, but bias ablation remains an unaddressed issue. We demonstrate a novel framework for inspecting bias in pre-trained transformer-based language models via movement pruning. Given a model and a debiasing objective, our framework finds a subset of the model containing less bias than the original model. We implement our framework by pruning the model while fine-tuning it on the debiasing objective. Optimized are only the pruning scores - parameters coupled with the model's weights that act as gates. We experiment with pruning attention heads, an important building block of transformers: we prune square blocks, as well as establish a new way of pruning the entire heads. Lastly, we demonstrate the usage of our framework using gender bias, and based on our findings, we propose an improvement to an existing debiasing method. Additionally, we re-discover a bias-performance trade-off: the better the model performs, the more bias it contains.