mlsanityData to 7 Oct 2026

Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons

arXiv 2403.07688 ↗First posted 12 Mar 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

When training neural networks, dying neurons -- units becoming inactive or saturated -- are traditionally seen as harmful. This paper sheds new light on this phenomenon. By exploring the impact of various hyperparameter configurations on dying neurons during training, we gather insights on how to improve upon sparse training approaches to pruning. We introduce Demon Pruning (DemP), a method that controls the proliferation of dead neurons through a combination of noise injection on active units and a one-cycle schedule regularization strategy, dynamically leading to network sparsity. Experiments on CIFAR-10 and ImageNet datasets demonstrate that DemP outperforms existing dense-to-sparse structured pruning methods, achieving better accuracy-sparsity tradeoffs and accelerating training by up to 3.56×. These findings provide a novel perspective on dying neurons as a resource for efficient model compression and optimization.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 61 of its 89 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

61 of 89 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
38 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  8. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  9. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  10. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  12. 2019The State of Sparsity in Deep Neural Networkscited by 471
  13. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  14. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  15. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  16. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  17. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  18. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  19. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  20. 2023Understanding plasticity in neural networkscited by 170
  21. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  22. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  23. 2020Sparse GPU Kernels for Deep Learningcited by 138
  24. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  25. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  26. 2019Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networkscited by 118
  27. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  28. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  29. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  30. 2020Neural Pruning via Growing Regularizationcited by 97
  31. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  32. 2019Fast Sparse ConvNetscited by 86
  33. 2015Fast ConvNets Using Group-wise Brain Damagecited by 81
  34. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  35. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  36. 2021A study on the plasticity of neural networkscited by 44
  37. 2019PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfigurationcited by 41
  38. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Adaptive Width Neural Networkscited by 4
  2. 2025Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptationcited by 3
  3. 2025Fairness Evaluation and Inference Level Mitigation in LLMscited by 2
  4. 2026Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generationcited by 2
  5. 2024Scalable iterative pruning of large language and vision models using block coordinate descentcited by 1