mlsanityData to 14 Oct 2026

The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth

arXiv 2404.18949 ↗First posted 27 Apr 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

While deep neural networks are highly effective at solving complex tasks, large pre-trained models are commonly employed even to solve consistently simpler downstream tasks, which do not necessarily require a large model's complexity. Motivated by the awareness of the ever-growing AI environmental impact, we propose an efficiency strategy that leverages prior knowledge transferred by large models. Simple but effective, we propose a method relying on an Entropy-bASed Importance mEtRic (EASIER) to reduce the depth of over-parametrized deep neural networks, which alleviates their computational burden. We assess the effectiveness of our method on traditional image classification setups. Our code is available at https://github.com/VGCQ/EASIER.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
25 more
  1. 2020In Search of Lost Domain Generalizationcited by 1,054
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  5. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  7. 2020What is the State of Neural Network Pruning?cited by 583
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  9. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  10. 2019The State of Sparsity in Deep Neural Networkscited by 476
  11. 2015Illuminating search spaces by mapping elitescited by 425
  12. 2017Large-Scale Evolution of Image Classifierscited by 348
  13. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  14. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  15. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  16. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 72
  17. 2021DeepReDuce: ReLU Reduction for Fast Private Inferencecited by 50
  18. 2022Selective Network Linearization for Efficient Private Inferencecited by 31
  19. 2023Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?cited by 20
  20. 2020LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networkscited by 19
  21. 2021Layer Folding: Neural Network Depth Reduction using Activation Linearizationcited by 14
  22. 2021SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networkscited by 12
  23. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  24. 2022Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Thinkcited by 4
  25. 2021Rethinking the Misalignment Problem in Dense Object Detectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Progress Measures for Grokking on Real-world Taskscited by 8
  2. 2024LaCoOT: Layer Collapse through Optimal Transportcited by 1
  3. 2026From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategycited by 0