mlsanityData to 7 Oct 2026

An Information-Theoretic Justification for Model Pruning

arXiv 2102.08329 ↗First posted 16 Feb 2021cs.LG cs.IT eess.SPcited by 23

Not on the map. Why?

Abstract (arXiv)

We study the neural network (NN) compression problem, viewing the tension between the compression ratio and NN performance through the lens of rate-distortion theory. We choose a distortion metric that reflects the effect of NN compression on the model output and derive the tradeoff between rate (compression) and distortion. In addition to characterizing theoretical limits of NN compression, this formulation shows that pruning, implicitly or explicitly, must be a part of a good compression algorithm. This observation bridges a gap between parts of the literature pertaining to NN and data compression, respectively, providing insight into the empirical success of model pruning. Finally, we propose a novel pruning strategy derived from our information-theoretic formulation and show that it outperforms the relevant baselines on CIFAR-10 and ImageNet datasets.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 55 of its 92 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

55 of 92 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  11. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
38 more
  1. 2016End-to-end Optimized Image Compressioncited by 673
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  3. 2018Rethinking the Value of Network Pruningcited by 595
  4. 2020What is the State of Neural Network Pruning?cited by 578
  5. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  6. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  7. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  8. 2019The State of Sparsity in Deep Neural Networkscited by 471
  9. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  10. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  11. 2018Model compression via distillation and quantizationcited by 287
  12. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  13. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  14. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  15. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  16. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  17. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  18. 2016Ternary Weight Networkscited by 195
  19. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  20. 2020Characterising Bias in Compressed Modelscited by 162
  21. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  22. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  23. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  24. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  25. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  26. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122
  27. 2017Bayesian Compression for Deep Learningcited by 116
  28. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  29. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  30. 2019Fast Sparse ConvNetscited by 86
  31. 2020Lookahead: A Far-Sighted Alternative of Magnitude-based Pruningcited by 61
  32. 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
  33. 2018Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameterscited by 56
  34. 2018Private Model Compression via Knowledge Distillationcited by 37
  35. 2019Scalable Model Compression by Entropy Penalized Reparameterizationcited by 31
  36. 2020rTop-k: A Statistical Estimation Approach to Distributed SGDcited by 27
  37. 2019DeepCABAC: A Universal Compression Algorithm for Deep Neural Networkscited by 26
  38. 2015Training CNNs with Low-Rank Filters for Efficient Image Classificationcited by 25

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 9; 2024: 3; 2025: 2.

23 papers.

Sort:

  1. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  2. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 28
  3. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  4. 2021LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networkscited by 19
  5. 2023Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networkscited by 15
  6. 2024LPViT: Low-Power Semi-structured Pruning for Vision Transformerscited by 15
  7. 2023Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimationcited by 9
  8. 2023Adaptive Compression in Federated Learning via Side Informationcited by 6
  9. 2022Rate-Constrained Remote Contextual Banditscited by 5
  10. 2023High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curvecited by 5
  11. 2022ResFed: Communication Efficient Federated Learning by Transmitting Deep Compressed Residualscited by 4
  12. 2023Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneitycited by 4
11 more
  1. 2024Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacycited by 4
  2. 2024MCNC: Manifold-Constrained Reparameterization for Neural Compressioncited by 4
  3. 2025Choose Your Model Size: Any Compression of Large Language Models Without Re-Computationcited by 3
  4. 2023Gradient Coding with Iterative Block Leverage Score Samplingcited by 2
  5. 2022Lossy Compression of Noisy Data for Private and Data-Efficient Learningcited by 1
  6. 2022Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithmscited by 1
  7. 2023M22: A Communication-Efficient Algorithm for Federated Learning Inspired by Rate-Distortioncited by 1
  8. 2023Rotation Invariant Quantization for Model Compressioncited by 1
  9. 2021AirNet: Neural Network Transmission over the Aircited by 0
  10. 2022How to Attain Communication-Efficient DNN Training? Convert, Compress, Correctcited by 0
  11. 2022PRANC: Pseudo RAndom Networks for Compacting deep modelscited by 0