mlsanityData to 7 Oct 2026

Are Compressed Language Models Less Subgroup Robust?

arXiv 2403.17811 ↗First posted 26 Mar 2024cs.LG cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels and attributes of a dataset. In this paper, we investigate the effects of 18 different compression methods and settings on the subgroup robustness of BERT language models. We show that worst-group performance does not depend on model size alone, but also on the compression method used. Additionally, we find that model compression does not always worsen the performance on minority subgroups. Altogether, our analysis serves to further research into the subgroup robustness of model compression.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

19 of 24 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
7 more
  1. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  2. 2020Characterising Bias in Compressed Modelscited by 162
  3. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  4. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 48
  5. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  6. 2022Subgroup Robustness Grows On Trees: An Empirical Baseline Investigationcited by 17
  7. 2022Okapi: Generalising Better by Making Statistical Matches Matchcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
  2. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformerscited by 2
  3. 2026When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruningcited by 0