mlsanityData to 7 Oct 2026

Layer-wise Model Pruning based on Mutual Information

arXiv 2108.12594 ↗First posted 28 Aug 2021cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

The proposed pruning strategy offers merits over weight-based pruning techniques: (1) it avoids irregular memory access since representations and matrices can be squeezed into their smaller but dense counterparts, leading to greater speedup; (2) in a manner of top-down pruning, the proposed method operates from a more global perspective based on training signals in the top layer, and prunes each layer by propagating the effect of global signals through layers, leading to better performances at the same sparsity level. Extensive experiments show that at the same sparsity level, the proposed strategy offers both greater speedup and higher performances than weight-based pruning methods (e.g., magnitude pruning, movement pruning).

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 70 of its 94 references that are in the map. First posted on arXiv 28 Aug 2021.

Builds on

70 of 94 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  11. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  12. 2018MINE: Mutual Information Neural Estimationcited by 680
  13. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  14. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  15. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  16. 2018Rethinking the Value of Network Pruningcited by 595
  17. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  18. 2019On Variational Bounds of Mutual Informationcited by 550
  19. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  20. 2019Importance Estimation for Neural Network Pruningcited by 531
  21. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  22. 2019The State of Sparsity in Deep Neural Networkscited by 471
  23. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  24. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  25. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  26. 2016FastText.zip: Compressing text classification modelscited by 331
  27. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  28. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  29. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  30. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  31. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  32. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  33. 2020Nearest Neighbor Machine Translationcited by 209
  34. 2019Structured Pruning of Large Language Modelscited by 201
  35. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  36. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  37. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  38. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 2; 2024: 7; 2025: 6; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  2. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  3. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  4. 2023Fairness-Aware Structured Pruning in Transformerscited by 21
  5. 2024Persistent Topological Features in Large Language Modelscited by 11
  6. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  7. 2024Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Modelscited by 7
  8. 2025Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealingcited by 4
  9. 2025Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Modelcited by 4
  10. 2024How Redundant Is the Transformer Stack in Speech Representation Models?cited by 2
  11. 2025MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Samplingcited by 2
  12. 2026MI-PRUN: Optimize Large Language Model Pruning via Mutual Informationcited by 2
7 more
  1. 2026GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selectioncited by 2
  2. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  3. 2024Large Language Model Pruningcited by 1
  4. 2024MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruningcited by 1
  5. 2025PLayer-FL: A Principled Approach to Personalized Layer-wise Cross-Silo Federated Learningcited by 1
  6. 2021Pruning Pretrained Encoders with a Multitask Objectivecited by 0
  7. 2025FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Modelscited by 0