mlsanityData to 7 Oct 2026

Can Model Compression Improve NLP Fairness

arXiv 2201.08542 ↗First posted 21 Jan 2022cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Model compression techniques are receiving increasing attention; however, the effect of compression on model fairness is still under explored. This is the first paper to examine the effect of distillation and pruning on the toxicity and bias of generative language models. We test Knowledge Distillation and Pruning methods on the GPT2 model and found a consistent pattern of toxicity and bias reduction after model distillation; this result can be potentially interpreted by existing line of research which describes model compression as a regularization technique; our work not only serves as a reference for safe deployment of compressed models, but also extends the discussion of "compression as regularization" into the setting of neural LMs, and hints at the possibility of using compression to develop fairer models.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 21 Jan 2022.

Builds on

27 of 34 references.

References · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2019Are Sixteen Heads Really Better than One?cited by 803
  11. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
15 more
  1. 2020Recipes for building an open-domain chatbotcited by 593
  2. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  3. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  4. 2020Bag of Tricks for Adversarial Trainingcited by 199
  5. 2020Characterising Bias in Compressed Modelscited by 162
  6. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  7. 2019Adversarially Robust Distillationcited by 124
  8. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  9. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  10. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  11. 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
  12. 2021On the Effect of Pruning on Adversarial Robustnesscited by 23
  13. 2020Going Beyond Classification Accuracy Metrics in Model Compressioncited by 15
  14. 2019Membership Privacy for Machine Learning Models Through Knowledge Transfercited by 3
  15. 2019Noise as a Resource for Learning in Knowledge Distillationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 7; 2024: 4; 2025: 3.

18 papers.

Sort:

By area:

  1. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  2. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  3. 2022Pruning has a disparate impact on model accuracycited by 43
  4. 2023Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learningcited by 42
  5. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  6. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  7. 2024On the social bias of speech self-supervised modelscited by 16
  8. 2023Being Right for Whose Right Reasons?cited by 12
  9. 2024The Impact of Inference Acceleration on Bias of LLMscited by 9
  10. 2025Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretabilitycited by 6
  11. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  12. 2022A Fair Loss Function for Network Pruningcited by 4
6 more
  1. 2024An investigation of structures responsible for gender bias in BERT and DistilBERTcited by 3
  2. 2025How Quantization Shapes Bias in Large Language Modelscited by 3
  3. 2023Dual-Teacher De-biasing Distillation Framework for Multi-domain Fake News Detectioncited by 2
  4. 2023On The Fairness Impacts of Hardware Selection in Machine Learningcited by 2
  5. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  6. 2025Downsized and Compromised?: Assessing the Faithfulness of Model Compressioncited by 0