# An investigation of structures responsible for gender bias in BERT and DistilBERT

arXiv: https://arxiv.org/abs/2401.06495 · HTML: https://mlsanity.com/paper/2401.06495/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Characterising Bias in Compressed Models](https://mlsanity.com/paper/2010.03058/) (2020)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [Compression of Deep Learning Models for Text: A Survey](https://mlsanity.com/paper/2008.05221/) (2020)
- [Investigating Gender Bias in BERT](https://mlsanity.com/paper/2009.05021/) (2020)
- [What are the biases in my word embedding?](https://mlsanity.com/paper/1812.08769/) (2018)
- [Teacher's pet: understanding and mitigating biases in distillation](https://mlsanity.com/paper/2106.10494/) (2021)
- [Can Model Compression Improve NLP Fairness](https://mlsanity.com/paper/2201.08542/) (2022)
- [FairDistillation: Mitigating Stereotyping in Language Models](https://mlsanity.com/paper/2207.04546/) (2022)

## Built on by (most cited first, 3 of 3)

- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [No Free Lunch in Language Model Bias Mitigation? Targeted Bias Reduction Can Exacerbate Unmitigated LLM Biases](https://mlsanity.com/paper/2511.18635/) (2025)
- [Fair-GPTQ: Bias-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2509.15206/) (2025)

## Abstract (arXiv)

In recent years, large Transformer-based Pre-trained Language Models (PLM) have changed the Natural Language Processing (NLP) landscape, by pushing the performance boundaries of the state-of-the-art on a wide variety of tasks. However, this performance gain goes along with an increase in complexity, and as a result, the size of such models (up to billions of parameters) represents a constraint for their deployment on embedded devices or short-inference time tasks. To cope with this situation, compressed models emerged (e.g. DistilBERT), democratizing their usage in a growing number of applications that impact our daily lives. A crucial issue is the fairness of the predictions made by both PLMs and their distilled counterparts. In this paper, we propose an empirical exploration of this problem by formalizing two questions: (1) Can we identify the neural mechanism(s) responsible for gender bias in BERT (and by extension DistilBERT)? (2) Does distillation tend to accentuate or mitigate gender bias (e.g. is DistilBERT more prone to gender bias than its uncompressed version, BERT)? Our findings are the following: (I) one cannot identify a specific layer that produces bias; (II) every attention head uniformly encodes bias; except in the context of underrepresented classes with a high imbalance of the sensitive attribute; (III) this subset of heads is different as we re-fine tune the network; (IV) bias is more homogeneously produced by the heads in the distilled model.