mlsanityData to 7 Oct 2026

Learning both Weights and Connections for Efficient Neural Networks

arXiv 1506.02626 ↗First posted 8 Jun 2015cs.NE cs.CV cs.LGcited by 1,998

Not on the map. Why?

Cited by 34 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9x, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the number of parameters can be reduced by 13x, from 138 million to 10.3 million, again with no loss of accuracy.

Key facts. Cited by 1,998 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 383; 2022: 334; 2023: 358; 2024: 337; 2025: 349; 2026: 236.

1,998 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  11. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
38 more
  1. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  4. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  5. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  6. 2021Accelerating Sparse Deep Neural Networkscited by 213
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  9. 2023DepGraph: Towards Any Structural Pruningcited by 202
  10. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  11. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  12. 2023Continual Pre-training of Language Modelscited by 194
  13. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  14. 2023EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingScited by 191
  15. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  16. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  17. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  18. 2021Block Pruning For Faster Transformerscited by 176
  19. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  20. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  21. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  22. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  23. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  24. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  25. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  26. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  27. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  28. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  29. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  30. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  31. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  32. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  33. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  34. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  35. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  36. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  37. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  38. 2023HNeRV: A Hybrid Neural Representation for Videoscited by 100

50 most cited of 1,998 All 1,998 (JSON)