# FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders

arXiv: https://arxiv.org/abs/2103.06413 · HTML: https://mlsanity.com/paper/2103.06413/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)

## Built on by (most cited first, 30 of 75)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Learning Debiased Representation via Disentangled Feature Augmentation](https://mlsanity.com/paper/2107.01372/) (2021)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Fairness via Representation Neutralization](https://mlsanity.com/paper/2106.12674/) (2021)
- [Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes](https://mlsanity.com/paper/2402.01981/) (2024)
- [FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering](https://mlsanity.com/paper/2504.14492/) (2025)
- [Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting](https://mlsanity.com/paper/2110.05367/) (2021)
- [Learning Fair Representation via Distributional Contrastive Disentanglement](https://mlsanity.com/paper/2206.08743/) (2022)
- [MABEL: Attenuating Gender Bias using Textual Entailment Data](https://mlsanity.com/paper/2210.14975/) (2022)
- [ToViLaG: Your Visual-Language Generative Model is Also An Evildoer](https://mlsanity.com/paper/2312.11523/) (2023)
- [Survey of Social Bias in Vision-Language Models](https://mlsanity.com/paper/2309.14381/) (2023)
- [Parameter-efficient Modularised Bias Mitigation via AdapterFusion](https://mlsanity.com/paper/2302.06321/) (2023)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases](https://mlsanity.com/paper/2307.01595/) (2023)
- [Mitigating Test-Time Bias for Fair Image Retrieval](https://mlsanity.com/paper/2305.19329/) (2023)
- [Joint Vision-Language Social Bias Removal for CLIP](https://mlsanity.com/paper/2411.12785/) (2024)
- [Modeling Techniques for Machine Learning Fairness: A Survey](https://mlsanity.com/paper/2111.03015/) (2021)
- [An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models](https://mlsanity.com/paper/2306.04067/) (2023)
- [Large Language Model Bias Mitigation from the Perspective of Knowledge Editing](https://mlsanity.com/paper/2405.09341/) (2024)
- [Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach](https://mlsanity.com/paper/2502.00577/) (2025)
- [Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning](https://mlsanity.com/paper/2310.12490/) (2023)
- [DebGCD: Debiased Learning with Distribution Guidance for Generalized Category Discovery](https://mlsanity.com/paper/2504.04804/) (2025)
- [Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads](https://mlsanity.com/paper/2311.10395/) (2023)
- [Identifying and Mitigating Social Bias Knowledge in Language Models](https://mlsanity.com/paper/2408.11843/) (2024)
- [Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Sampling](https://mlsanity.com/paper/2302.13136/) (2023)
- [Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models](https://mlsanity.com/paper/2307.10522/) (2023)
- [Conditional Supervised Contrastive Learning for Fair Text Classification](https://mlsanity.com/paper/2205.11485/) (2022)

## Abstract (arXiv)

Pretrained text encoders, such as BERT, have been applied increasingly in various natural language processing (NLP) tasks, and have recently demonstrated significant performance gains. However, recent studies have demonstrated the existence of social bias in these pretrained NLP models. Although prior works have made progress on word-level debiasing, improved sentence-level fairness of pretrained encoders still lacks exploration. In this paper, we proposed the first neural debiasing method for a pretrained sentence encoder, which transforms the pretrained encoder outputs into debiased representations via a fair filter (FairFil) network. To learn the FairFil, we introduce a contrastive learning framework that not only minimizes the correlation between filtered embeddings and bias words but also preserves rich semantic information of the original sentences. On real-world datasets, our FairFil effectively reduces the bias degree of pretrained text encoders, while continuously showing desirable performance on downstream tasks. Moreover, our post-hoc method does not require any retraining of the text encoders, further enlarging FairFil's application space.