mlsanityData to 7 Oct 2026

FairDistillation: Mitigating Stereotyping in Language Models

arXiv 2207.04546 ↗First posted 10 Jul 2022cs.CL cs.CY cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Large pre-trained language models are successfully being used in a variety of tasks, across many languages. With this ever-increasing usage, the risk of harmful side effects also rises, for example by reproducing and reinforcing stereotypes. However, detecting and mitigating these harms is difficult to do in general and becomes computationally expensive when tackling multiple languages or when considering different biases. To address this, we present FairDistillation: a cross-lingual method based on knowledge distillation to construct smaller language models while controlling for specific biases. We found that our distillation method does not negatively affect the downstream performance on most tasks and successfully mitigates stereotyping and representational harms. We demonstrate that FairDistillation can create fairer language models at a considerably lower cost than alternative approaches.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 10 Jul 2022.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
18 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  3. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  4. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  5. 2019CamemBERT: a Tasty French Language Modelcited by 368
  6. 2019Measuring Bias in Contextualized Word Representationscited by 275
  7. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  8. 2018Gender Bias in Neural Natural Language Processingcited by 210
  9. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  10. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  11. 2019It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitutioncited by 102
  12. 2021Sustainable Modular Debiasing of Language Modelscited by 98
  13. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  14. 2020Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Biascited by 82
  15. 2022AugLy: Data Augmentations for Robustnesscited by 34
  16. 2021Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Modelscited by 20
  17. 2019The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviewscited by 14
  18. 2022RobBERTje: a Distilled Dutch BERT Modelcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 1; 2026: 1.

8 papers.

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2023A Survey on Fairness in Large Language Modelscited by 91
  3. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  4. 2024Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigationcited by 17
  5. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  6. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  7. 2024An investigation of structures responsible for gender bias in BERT and DistilBERTcited by 3
  8. 2026Bias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Modelscited by 0