mlsanityData to 7 Oct 2026

Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding

arXiv 2110.08419 ↗First posted 16 Oct 2021cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Recent work has focused on compressing pre-trained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain similar performance on in-distribution development sets for a task. Further analysis indicates that the compressed models overfit on the shortcut samples and generalize poorly on the hard ones. We further leverage this observation to develop a regularization strategy for robust model compression based on sample uncertainty. Experimental results on several natural language understanding tasks demonstrate that our bias mitigation framework improves the OOD generalization of the compressed models, while not sacrificing the in-distribution task performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 48 of its 56 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

48 of 56 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2019When Does Label Smoothing Help?cited by 985
36 more
  1. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  7. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  8. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  9. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  10. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  11. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  12. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  13. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  14. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  15. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  16. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  17. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  18. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  19. 2020Characterising Bias in Compressed Modelscited by 162
  20. 2019Towards Debiasing Fact Verification Modelscited by 143
  21. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  22. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  23. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  24. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  25. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  26. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  27. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  28. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  29. 2018Did the Model Understand the Question?cited by 60
  30. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  31. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  32. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  33. 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
  34. 2020Posterior Differential Regularization with f-divergence for Improving Model Robustnesscited by 20
  35. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 20
  36. 2019Increasing Robustness to Spurious Correlations using Forgettable Examplescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 5; 2025: 2; 2026: 1.

12 papers.

  1. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  2. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  3. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  4. 2024On the Challenges and Opportunities in Generative AIcited by 31
  5. 2024Can LLMs make trade-offs involving stipulated pain and pleasure states?cited by 16
  6. 2024Pruning as a Domain-specific LLM Extractorcited by 12
  7. 2025The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperationcited by 11
  8. 2024Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retrainingcited by 8
  9. 2023Using Early Readouts to Mediate Featural Bias in Distillationcited by 7
  10. 2025A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?cited by 1
  11. 2024Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understandingcited by 0
  12. 2026Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Yearcited by 0