mlsanityData to 7 Oct 2026

Learning the Number of Neurons in Deep Networks

arXiv 1611.06321 ↗First posted 19 Nov 2016cs.CV cs.LG cs.NEcited by 69

Not on the map. Why?

Abstract (arXiv)

Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of structured sparsity during learning. More precisely, we use a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80% while retaining or even improving the network accuracy.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 27; 2022: 17; 2023: 12; 2024: 7; 2025: 6.

69 papers.

Sort:

By area:

  1. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  2. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 66
  3. 2022Online Continual Learning for Embedded Devicescited by 48
  4. 2022Structural Pruning via Latency-Saliency Knapsackcited by 40
  5. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  6. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  7. 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
  8. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  9. 2021ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximationscited by 21
  10. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  11. 2021Stealthy Backdoors as Compression Artifactscited by 17
  12. 2021Neural Network Training Using ℓ1-Regularization and Bi-fidelity Datacited by 16
38 more
  1. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
  2. 2021Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Detailscited by 14
  3. 2021Convolutional Neural Network Compression through Generalized Kronecker Product Decompositioncited by 14
  4. 2022Provable and Efficient Continual Representation Learningcited by 14
  5. 2022Soft Masking for Cost-Constrained Channel Pruningcited by 12
  6. 2021Machine Learning for the Detection and Identification of Internet of Things (IoT) Devices: A Surveycited by 9
  7. 2021Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsitycited by 9
  8. 2021Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNNcited by 9
  9. 2022Sequential Attention for Feature Selectioncited by 9
  10. 2022Statistical guarantees for sparse deep learningcited by 9
  11. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  12. 2021HALP: Hardware-Aware Latency Pruningcited by 8
  13. 2021Deep Model Compression based on the Training Historycited by 7
  14. 2021Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networkscited by 5
  15. 2021Distilling Image Classifiers in Object Detectorscited by 5
  16. 2022Pruning-aware Sparse Regularization for Network Pruningcited by 5
  17. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  18. 2023Magnitude Attention-based Dynamic Pruningcited by 5
  19. 2021AACP: Model Compression by Accurate and Automatic Channel Pruningcited by 4
  20. 2023Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networkscited by 4
  21. 2024Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraintcited by 4
  22. 2025MDP: Multidimensional Vision Model Pruning with Latency Constraintcited by 4
  23. 2021Alternate Model Growth and Pruning for Efficient Training of Recommendation Systemscited by 3
  24. 2021Content-Aware Convolutional Neural Networkscited by 3
  25. 2022Entropy Induced Pruning Framework for Convolutional Neural Networkscited by 3
  26. 2022Learning sparse auto-encoders for green AI image codingcited by 3
  27. 2023Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Ticketscited by 3
  28. 2024DRIVE: Dual Gradient-Based Rapid Iterative Pruningcited by 3
  29. 2022Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learningcited by 2
  30. 2022Scaling Private Deep Learning with Low-Rank and Sparse Gradientscited by 2
  31. 2022MGiaD: Multigrid in all dimensions. Efficiency and robustness by coarsening in resolution and channel dimensionscited by 2
  32. 2023AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentationcited by 2
  33. 2024Enhanced Sparsification via Stimulative Trainingcited by 2
  34. 2025Regularization can make diffusion models more efficientcited by 2
  35. 2021Reframing Neural Networks: Deep Structure in Overcomplete Representationscited by 1
  36. 2021Schematic Memory Persistence and Transience for Efficient and Robust Continual Learningcited by 1
  37. 2021Lockout: Sparse Regularization of Neural Networkscited by 1
  38. 2021Searching for Minimal Optimal Neural Networkscited by 1

50 most cited of 69 All 69 (JSON)