mlsanityData to 14 Oct 2026

Understanding the Effect of Model Compression on Social Bias in Large Language Models

arXiv 2312.05662 ↗First posted 9 Dec 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tasks, leading to representational harm. Many strategies have been proposed to mitigate the effects of inappropriate social biases learned during pretraining. Simultaneously, methods for model compression have become increasingly popular to reduce the computational burden of LLMs. Despite the popularity and need for both approaches, little work has been done to explore the interplay between these two. We perform a carefully controlled study of the impact of model compression via quantization and knowledge distillation on measures of social bias in LLMs. Longer pretraining and larger models led to higher social bias, and quantization showed a regularizer effect with its best trade-off around 20% of the original pretraining time.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

26 of 37 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  9. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  10. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  11. 2019Measuring Bias in Contextualized Word Representationscited by 276
  12. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
14 more
  1. 2019Depth-Adaptive Transformercited by 236
  2. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  3. 2019What Do Compressed Deep Neural Networks Forget?cited by 171
  4. 2020Towards Debiasing Sentence Representationscited by 171
  5. 2020Characterising Bias in Compressed Modelscited by 167
  6. 2021Debiasing Pre-trained Contextualised Embeddingscited by 107
  7. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  8. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  9. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
  10. 2023Intriguing Properties of Quantization at Scalecited by 49
  11. 2022Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Taskscited by 48
  12. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
  13. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  14. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024Building Guardrails for Large Language Modelscited by 65
  2. 2024On the social bias of speech self-supervised modelscited by 16
  3. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 14
  4. 2025What's Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLMscited by 8
  5. 2025FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Modelscited by 8
  6. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levelscited by 6
  7. 2024Model Compression Techniques in Biometrics Applications: A Surveycited by 5
  8. 2024Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibrationcited by 5
  9. 2026Investigating Social Bias Changes in Quantized Language Modelscited by 5
  10. 2025How Quantization Shapes Bias in Large Language Modelscited by 4
  11. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 3
  12. 2024Surgical Feature-Space Decomposition of LLMs: Why, When and How?cited by 2
7 more
  1. 2025Investigating Hallucination in Conversations for Low Resource Languagescited by 2
  2. 2026Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protectioncited by 1
  3. 2026Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusionscited by 1
  4. 2025Fair-GPTQ: Bias-Aware Quantization for Large Language Modelscited by 0
  5. 2025Debiasing Large Language Models in Thai Political Stance Detection via Counterfactual Calibrationcited by 0
  6. 2025Downsized and Compromised?: Assessing the Faithfulness of Model Compressioncited by 0
  7. 2026Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models
  2. 2025An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation
  3. 2026Evaluating Social Bias in RAG Systems: When External Context Helps and Reasoning Hurts
  4. 2024Interpreting Bias in Large Language Models: A Feature-Based Approach
  5. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression
  6. 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection
  7. 2026QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
  8. 2023SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models