mlsanityData to 14 Oct 2026

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

arXiv 1510.00149 ↗First posted 1 Oct 2015cs.CV cs.NEcited by 2,329

Not on the map. Why?

Cited by 84 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.

Key facts. Cited by 2,329 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Oct 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 490; 2022: 385; 2023: 402; 2024: 379; 2025: 380; 2026: 292.

2,329 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  9. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  12. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
38 more
  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  2. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 386
  3. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  4. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  5. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  8. 2021Post-Training Quantization for Vision Transformercited by 278
  9. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  10. 2023A Survey on Model Compression for Large Language Modelscited by 269
  11. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  12. 2021NeRV: Neural Representations for Videoscited by 256
  13. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  14. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  15. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 234
  16. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  17. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  18. 2023Model Sparsity Can Simplify Machine Unlearningcited by 214
  19. 2023DepGraph: Towards Any Structural Pruningcited by 203
  20. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  21. 2022Medical Image Segmentation Review: The success of U-Netcited by 191
  22. 2021Does Knowledge Distillation Really Work?cited by 190
  23. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  24. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  25. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  26. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  27. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  28. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 175
  29. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 168
  30. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  31. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  32. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  33. 2021Implicit Neural Representations for Image Compressioncited by 162
  34. 2022On-Device Training Under 256KB Memorycited by 160
  35. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
  36. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  37. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  38. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142

50 most cited of 2,329 All 2,329 (JSON)