mlsanityData to 7 Oct 2026

FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders

arXiv 2103.06413 ↗First posted 11 Mar 2021cs.CL cs.LGcited by 75

Not on the map. Why?

Abstract (arXiv)

Pretrained text encoders, such as BERT, have been applied increasingly in various natural language processing (NLP) tasks, and have recently demonstrated significant performance gains. However, recent studies have demonstrated the existence of social bias in these pretrained NLP models. Although prior works have made progress on word-level debiasing, improved sentence-level fairness of pretrained encoders still lacks exploration. In this paper, we proposed the first neural debiasing method for a pretrained sentence encoder, which transforms the pretrained encoder outputs into debiased representations via a fair filter (FairFil) network. To learn the FairFil, we introduce a contrastive learning framework that not only minimizes the correlation between filtered embeddings and bias words but also preserves rich semantic information of the original sentences. On real-world datasets, our FairFil effectively reduces the bias degree of pretrained text encoders, while continuously showing desirable performance on downstream tasks. Moreover, our post-hoc method does not require any retraining of the text encoders, further enlarging FairFil's application space.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

43 of 56 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
31 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  3. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2016Deep Variational Information Bottleneckcited by 915
  8. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  9. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  12. 2015LINE: Large-scale Information Network Embeddingcited by 582
  13. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  14. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  15. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  16. 2015Skip-Thought Vectorscited by 321
  17. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  18. 2019Measuring Bias in Contextualized Word Representationscited by 275
  19. 2019Evaluating Gender Bias in Machine Translationcited by 247
  20. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  21. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  22. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  23. 2020Towards Debiasing Sentence Representationscited by 170
  24. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  25. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  26. 2018Learning Controllable Fair Representationscited by 83
  27. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  28. 2018Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approachcited by 63
  29. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61
  30. 2020Improving Disentangled Text Representation Learning with Information-Theoretic Guidancecited by 52
  31. 2019Learning Compressed Sentence Representations for On-Device Text Processingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 25; 2024: 14; 2025: 14; 2026: 4.

75 papers.

Sort:

By area:

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2021Learning Debiased Representation via Disentangled Feature Augmentationcited by 116
  3. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  4. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  5. 2023A Survey on Fairness in Large Language Modelscited by 91
  6. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  7. 2021Fairness via Representation Neutralizationcited by 63
  8. 2024Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypescited by 58
  9. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 37
  10. 2021Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgettingcited by 34
  11. 2022Learning Fair Representation via Distributional Contrastive Disentanglementcited by 31
  12. 2022MABEL: Attenuating Gender Bias using Textual Entailment Datacited by 31
38 more
  1. 2023ToViLaG: Your Visual-Language Generative Model is Also An Evildoercited by 27
  2. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  3. 2023Parameter-efficient Modularised Bias Mitigation via AdapterFusioncited by 25
  4. 2022Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Surveycited by 22
  5. 2023Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biasescited by 21
  6. 2023Mitigating Test-Time Bias for Fair Image Retrievalcited by 20
  7. 2021Modeling Techniques for Machine Learning Fairness: A Surveycited by 17
  8. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 17
  9. 2023An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Modelscited by 16
  10. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  11. 2023Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuningcited by 15
  12. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  13. 2025DebGCD: Debiased Learning with Distribution Guidance for Generalized Category Discoverycited by 14
  14. 2023Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Headscited by 13
  15. 2024Identifying and Mitigating Social Bias Knowledge in Language Modelscited by 13
  16. 2023Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Samplingcited by 12
  17. 2023Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Modelscited by 12
  18. 2022Conditional Supervised Contrastive Learning for Fair Text Classificationcited by 11
  19. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  20. 2023Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translationcited by 10
  21. 2023Unpacking the Ethical Value Alignment in Big Modelscited by 9
  22. 2023Travel Demand Forecasting: A Fair AI Approachcited by 7
  23. 2023Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Toolscited by 7
  24. 2024Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Modelscited by 7
  25. 2024AXOLOTL: Fairness through Assisted Self-Debiasing of Large Language Model Outputscited by 7
  26. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  27. 2022Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Informationcited by 5
  28. 2022Sustaining Fairness via Incremental Learningcited by 5
  29. 2023ENGAGE: Explanation Guided Data Augmentation for Graph Representation Learningcited by 5
  30. 2025Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentencescited by 5
  31. 2023Fair Text Classification with Wasserstein Independencecited by 4
  32. 2025ConFit v2: Improving Resume-Job Matching using Hypothetical Resume Embedding and Runner-Up Hard-Negative Miningcited by 4
  33. 2024Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularizationcited by 3
  34. 2025Fair Text Classification via Transferable Representationscited by 3
  35. 2025Mitigating Confounding in Speech-Based Dementia Detection through Weight Maskingcited by 3
  36. 2025KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Modelscited by 3
  37. 2023Pushing the Accuracy-Group Robustness Frontier with Introspective Self-playcited by 2
  38. 2023General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Levelcited by 2

50 most cited of 75 All 75 (JSON)