# Gender-preserving Debiasing for Pre-trained Word Embeddings

arXiv: https://arxiv.org/abs/1906.00742 · HTML: https://mlsanity.com/paper/1906.00742/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 30 of 61)

- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders](https://mlsanity.com/paper/2103.06413/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Unmasking the Mask -- Evaluating Social Biases in Masked Language Models](https://mlsanity.com/paper/2104.07496/) (2021)
- [Gender Bias in Masked Language Models for Multiple Languages](https://mlsanity.com/paper/2205.00551/) (2022)
- [Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks](https://mlsanity.com/paper/2210.02938/) (2022)
- [Generalizing to the Future: Mitigating Entity Bias in Fake News Detection](https://mlsanity.com/paper/2204.09484/) (2022)
- [Choose Your Lenses: Flaws in Gender Bias Evaluation](https://mlsanity.com/paper/2210.11471/) (2022)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Survey of Social Bias in Vision-Language Models](https://mlsanity.com/paper/2309.14381/) (2023)
- [Dictionary-based Debiasing of Pre-trained Word Embeddings](https://mlsanity.com/paper/2101.09525/) (2021)
- [Intersectional Bias in Causal Language Models](https://mlsanity.com/paper/2107.07691/) (2021)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning](https://mlsanity.com/paper/2207.02463/) (2022)
- [ADEPT: A DEbiasing PrompT Framework](https://mlsanity.com/paper/2211.05414/) (2022)
- [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022)
- [Obtaining Better Static Word Embeddings Using Contextual Embedding Models](https://mlsanity.com/paper/2106.04302/) (2021)
- [Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings](https://mlsanity.com/paper/2203.07523/) (2022)
- [How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processing](https://mlsanity.com/paper/2205.10095/) (2022)
- [Don't Forget About Pronouns: Removing Gender Bias in Language Models Without Losing Factual Gender Information](https://mlsanity.com/paper/2206.10744/) (2022)
- [Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads](https://mlsanity.com/paper/2311.10395/) (2023)
- [Style Pooling: Automatic Text Style Obfuscation for Improved Classification Fairness](https://mlsanity.com/paper/2109.04624/) (2021)
- [You Are What You Write: Preserving Privacy in the Era of Large Language Models](https://mlsanity.com/paper/2204.09391/) (2022)
- [Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models](https://mlsanity.com/paper/2307.10522/) (2023)
- [In-Contextual Gender Bias Suppression for Large Language Models](https://mlsanity.com/paper/2309.07251/) (2023)
- [Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach](https://mlsanity.com/paper/2412.11679/) (2024)
- [Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving](https://mlsanity.com/paper/2112.05194/) (2021)
- [Combining Static and Contextualised Multilingual Embeddings](https://mlsanity.com/paper/2203.09326/) (2022)

## Abstract (arXiv)

Word embeddings learnt from massive text collections have demonstrated significant levels of discriminative biases such as gender, racial or ethnic biases, which in turn bias the down-stream NLP applications that use those word embeddings. Taking gender-bias as a working example, we propose a debiasing method that preserves non-discriminative gender-related information, while removing stereotypical discriminative gender biases from pre-trained word embeddings. Specifically, we consider four types of information: feminine, masculine, gender-neutral and stereotypical, which represent the relationship between gender vs. bias, and propose a debiasing method that (a) preserves the gender-related information in feminine and masculine words, (b) preserves the neutrality in gender-neutral words, and (c) removes the biases from stereotypical words. Experimental results on several previously proposed benchmark datasets show that our proposed method can debias pre-trained word embeddings better than existing SoTA methods proposed for debiasing word embeddings while preserving gender-related but non-discriminative information.