mlsanityData to 7 Oct 2026

Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression

arXiv 2109.03228 ↗First posted 7 Sep 2021cs.CL cs.AI cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent studies on compression of pretrained language models (e.g., BERT) usually use preserved accuracy as the metric for evaluation. In this paper, we propose two new metrics, label loyalty and probability loyalty that measure how closely a compressed model (i.e., student) mimics the original model (i.e., teacher). We also explore the effect of compression with regard to robustness under adversarial attacks. We benchmark quantization, pruning, knowledge distillation and progressive module replacing with loyalty and robustness. By combining multiple compression techniques, we provide a practical strategy to achieve better accuracy, loyalty and robustness.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 7 Sep 2021.

Builds on

34 of 45 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2016Adversarial examples in the physical worldcited by 1,762
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
22 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2019Green AIcited by 620
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  8. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  9. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  10. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  11. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  12. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  13. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  14. 2020Characterising Bias in Compressed Modelscited by 162
  15. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 151
  16. 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
  17. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  18. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  19. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  20. 2019Detecting Overfitting via Adversarial Examplescited by 15
  21. 2020Quantisation and Pruning for Neural Network Compression and Regularisationcited by 9
  22. 2020SOAR: Second-Order Adversarial Regularizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 20; 2023: 5; 2024: 3; 2025: 5; 2026: 2.

38 papers.

Sort:

By area:

  1. 2023A Survey on Model Compression for Large Language Modelscited by 263
  2. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  3. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  4. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  5. 2021A Survey on Green Deep Learningcited by 52
  6. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  7. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  8. 2022Pruning has a disparate impact on model accuracycited by 43
  9. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  10. 2024Accuracy is Not All You Needcited by 27
  11. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  12. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
26 more
  1. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  2. 2022Robust Lottery Tickets for Pre-trained Language Modelscited by 16
  3. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  4. 2024Superficial Safety Alignment Hypothesiscited by 14
  5. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teacherscited by 13
  6. 2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12
  7. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
  8. 2023Bias in Pruned Vision Models: In-Depth Analysis and Countermeasurescited by 11
  9. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  10. 2022Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithmcited by 9
  11. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9
  12. 2024Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retrainingcited by 8
  13. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  14. 2023Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Modelscited by 7
  15. 2022Feature Structure Distillation with Centered Kernel Alignment in BERT Transferringcited by 6
  16. 2022Robust Distillation for Worst-class Performancecited by 6
  17. 2022Gradient Knowledge Distillation for Pre-trained Language Modelscited by 5
  18. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  19. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  20. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Modelscited by 3
  21. 2025A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?cited by 1
  22. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answeringcited by 0
  23. 2022Improving Imbalanced Text Classification with Dynamic Curriculum Learningcited by 0
  24. 2025As easy as PIE: understanding when pruning causes language models to disagreecited by 0
  25. 2026On Measuring Semantic Preservation in Legal Ontology Learningcited by 0
  26. 2026How Divergence Becomes Decision Flips in Compressed Language Modelscited by 0