mlsanityData to 7 Oct 2026

Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models

arXiv 2307.10522 ↗First posted 20 Jul 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora. Existing solutions require debiasing training processes and datasets for debiasing, which are resource-intensive and costly. Furthermore, these methods hurt the PLMs' performance on downstream tasks. In this study, we propose Gender-tuning, which debiases the PLMs through fine-tuning on downstream tasks' datasets. For this aim, Gender-tuning integrates Masked Language Modeling (MLM) training objectives into fine-tuning's training process. Comprehensive experiments show that Gender-tuning outperforms the state-of-the-art baselines in terms of average gender bias scores in PLMs while improving PLMs' performance on downstream tasks solely using the downstream tasks' dataset. Also, Gender-tuning is a deployable debiasing tool for any PLM that works with original fine-tuning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

27 of 38 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2018Neural Network Acceptability Judgmentscited by 952
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  11. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  12. 2019On Measuring Social Biases in Sentence Encoderscited by 397
15 more
  1. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  2. 2019Measuring Bias in Contextualized Word Representationscited by 275
  3. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  4. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  5. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  6. 2020Towards Debiasing Sentence Representationscited by 170
  7. 2018Learning Gender-Neutral Word Embeddingscited by 169
  8. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  9. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  10. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  11. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  12. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  13. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  14. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  15. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 3.

12 papers.

  1. 2023A Survey on Fairness in Large Language Modelscited by 91
  2. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  3. 2025Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editingcited by 14
  4. 2024AGR: Age Group fairness Reward for Bias Mitigation in LLMscited by 12
  5. 2025Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentencescited by 5
  6. 2024The Life Cycle of Large Language Models: A Review of Biases in Educationcited by 3
  7. 2024Expert-Guided Extinction of Toxic Tokens for Debiased Generationcited by 1
  8. 2025Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervisioncited by 0
  9. 2025Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluationcited by 0
  10. 2026FairNVT: Fair Classification via Noise Injection in Vision Transformerscited by 0
  11. 2026Neutrality Bites: Gender Representation in AI-Generated Animal Storiescited by 0
  12. 2026Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decodingcited by 0