# Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models

arXiv: https://arxiv.org/abs/2307.10522 · HTML: https://mlsanity.com/paper/2307.10522/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://mlsanity.com/paper/2010.06032/) (2020)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)
- [Understanding the Origins of Bias in Word Embeddings](https://mlsanity.com/paper/1810.03611/) (2018)
- [FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders](https://mlsanity.com/paper/2103.06413/) (2021)
- [Gender-preserving Debiasing for Pre-trained Word Embeddings](https://mlsanity.com/paper/1906.00742/) (2019)

## Built on by (most cited first, 12 of 12)

- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing](https://mlsanity.com/paper/2501.14457/) (2025)
- [AGR: Age Group fairness Reward for Bias Mitigation in LLMs](https://mlsanity.com/paper/2409.04340/) (2024)
- [Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences](https://mlsanity.com/paper/2501.06795/) (2025)
- [The Life Cycle of Large Language Models: A Review of Biases in Education](https://mlsanity.com/paper/2407.11203/) (2024)
- [Expert-Guided Extinction of Toxic Tokens for Debiased Generation](https://mlsanity.com/paper/2405.19299/) (2024)
- [Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision](https://mlsanity.com/paper/2508.09087/) (2025)
- [Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation](https://mlsanity.com/paper/2512.10734/) (2025)
- [FairNVT: Fair Classification via Noise Injection in Vision Transformers](https://mlsanity.com/paper/2604.16780/) (2026)
- [Neutrality Bites: Gender Representation in AI-Generated Animal Stories](https://mlsanity.com/paper/2606.07969/) (2026)
- [Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding](https://mlsanity.com/paper/2608.21415/) (2026)

## Abstract (arXiv)

Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora. Existing solutions require debiasing training processes and datasets for debiasing, which are resource-intensive and costly. Furthermore, these methods hurt the PLMs' performance on downstream tasks. In this study, we propose Gender-tuning, which debiases the PLMs through fine-tuning on downstream tasks' datasets. For this aim, Gender-tuning integrates Masked Language Modeling (MLM) training objectives into fine-tuning's training process. Comprehensive experiments show that Gender-tuning outperforms the state-of-the-art baselines in terms of average gender bias scores in PLMs while improving PLMs' performance on downstream tasks solely using the downstream tasks' dataset. Also, Gender-tuning is a deployable debiasing tool for any PLM that works with original fine-tuning.