mlsanityData to 7 Oct 2026

A Survey on Model Compression and Acceleration for Pretrained Language Models

arXiv 2202.07105 ↗First posted 15 Feb 2022cs.CL cs.AI cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Despite achieving state-of-the-art performance on many NLP tasks, the high energy cost and long inference delay prevent Transformer-based pretrained language models (PLMs) from seeing broader adoption including for edge and mobile computing. Efficient NLP research aims to comprehensively consider computation, time and carbon emission for the entire life-cycle of NLP, including data preparation, model training and inference. In this survey, we focus on the inference stage and review the current state of model compression and acceleration for pretrained language models, including benchmarks, metrics and methodology.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 87 of its 117 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

87 of 117 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2018Deep contextualized word representationscited by 2,285
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  11. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  12. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  13. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  14. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  15. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  16. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  17. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  18. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  19. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  20. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  21. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  22. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  23. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  24. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  25. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  26. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  27. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  28. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  29. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  30. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  31. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  32. 2021Does Knowledge Distillation Really Work?cited by 187
  33. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  34. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  35. 2021Block Pruning For Faster Transformerscited by 176
  36. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  37. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  38. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 13; 2025: 10; 2026: 8.

45 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  3. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  4. 2023ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Predictioncited by 27
  5. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  6. 2023Medical Vision Language Pretraining: A surveycited by 24
  7. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  8. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  9. 2024Enhancing Recommendation Diversity by Re-ranking with Large Language Modelscited by 19
  10. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  11. 2024The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Promptscited by 13
  12. 2025Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronoscited by 13
33 more
  1. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Modelscited by 12
  2. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  3. 2024CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelinescited by 11
  4. 2023Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroupcited by 8
  5. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  6. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  7. 2023Enhancing Interpretability using Human Similarity Judgements to Prune Word Embeddingscited by 6
  8. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  9. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 6
  10. 2022QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noisecited by 4
  11. 2024Harnessing Orthogonality to Train Low-Rank Neural Networkscited by 4
  12. 2025Theoretical Guarantees for Low-Rank Compression of Deep Neural Networkscited by 3
  13. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  14. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  15. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  16. 2025An Empirical Study of Knowledge Distillation for Code Understanding Taskscited by 2
  17. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 2
  18. 2023Orchestration of Emulator Assisted Mobile Edge Tuning for AI Foundation Models: A Multi-Agent Deep Reinforcement Learning Approachcited by 1
  19. 2025CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answeringcited by 1
  20. 2025StructPrune: Structured Global Pruning asymptotics with O(√N) GPU Memorycited by 1
  21. 2025A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?cited by 1
  22. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  23. 2023One-stop Training of Multiple Capacity Modelscited by 0
  24. 2023Generate to Understand for Representationcited by 0
  25. 2024Adaptive Inference: Theoretical Limits and Unexplored Opportunitiescited by 0
  26. 2024I-MPN: Inductive Message Passing Network for Efficient Human-in-the-Loop Annotation of Mobile Eye Tracking Datacited by 0
  27. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  28. 2025Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfactioncited by 0
  29. 2026How Is Uncertainty Propagated in Knowledge Distillation?cited by 0
  30. 2026Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compressioncited by 0
  31. 2026SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Modelscited by 0
  32. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
  33. 2026SurveyAgent-HKA: A multi-agent framework for scientific survey generation with LLMs and human knowledge augmentationcited by 0