# Group Sparse Regularization for Deep Neural Networks

arXiv: https://arxiv.org/abs/1607.00485 · HTML: https://mlsanity.com/paper/1607.00485/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2016.

## Built on by (most cited first, 30 of 89)

- [Sparse Low-rank Adaptation of Pre-trained Language Models](https://mlsanity.com/paper/2311.11696/) (2023)
- [ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models](https://mlsanity.com/paper/2402.13516/) (2024)
- [Locally Sparse Neural Networks for Tabular Biomedical Data](https://mlsanity.com/paper/2106.06468/) (2021)
- [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023)
- [DECORE: Deep Compression with Reinforcement Learning](https://mlsanity.com/paper/2106.06091/) (2021)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [Continual Learning of Diffusion Models with Generative Distillation](https://mlsanity.com/paper/2311.14028/) (2023)
- [Continual Graph Learning: A Survey](https://mlsanity.com/paper/2301.12230/) (2023)
- [spred: Solving L1 Penalty with SGD](https://mlsanity.com/paper/2210.01212/) (2022)
- [Neural Network Training Using ℓ1-Regularization and Bi-fidelity Data](https://mlsanity.com/paper/2105.13011/) (2021)
- [Learning Nonlinear Projections for Reduced-Order Modeling of Dynamical Systems using Constrained Autoencoders](https://mlsanity.com/paper/2307.15288/) (2023)
- [A Bregman Learning Framework for Sparse Neural Networks](https://mlsanity.com/paper/2105.04319/) (2021)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries](https://mlsanity.com/paper/2502.02496/) (2025)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [End-to-end learnable EEG channel selection for deep neural networks with Gumbel-softmax](https://mlsanity.com/paper/2102.09050/) (2021)
- [PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction](https://mlsanity.com/paper/2402.08198/) (2024)
- [Sequential Attention for Feature Selection](https://mlsanity.com/paper/2209.14881/) (2022)
- [Statistical guarantees for sparse deep learning](https://mlsanity.com/paper/2212.05427/) (2022)
- [Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood](https://mlsanity.com/paper/2402.15978/) (2024)
- [Deep Feature Screening: Feature Selection for Ultra High-Dimensional Data via Deep Neural Networks](https://mlsanity.com/paper/2204.01682/) (2022)
- [The Contextual Lasso: Sparse Linear Models via Deep Neural Networks](https://mlsanity.com/paper/2302.00878/) (2023)
- [Neural Architecture Search via Bregman Iterations](https://mlsanity.com/paper/2106.02479/) (2021)
- [An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity](https://mlsanity.com/paper/2205.05428/) (2022)
- [How a student becomes a teacher: learning and forgetting through Spectral methods](https://mlsanity.com/paper/2310.12612/) (2023)
- [Non-linear, Sparse Dimensionality Reduction via Path Lasso Penalized Autoencoders](https://mlsanity.com/paper/2102.10873/) (2021)
- [On the Compression of Neural Networks Using ℓ0-Norm Regularization and Weight Pruning](https://mlsanity.com/paper/2109.05075/) (2021)

## Abstract (arXiv)

In this paper, we consider the joint task of simultaneously optimizing (i) the weights of a deep neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are generally dealt with separately, we present a simple regularized formulation allowing to solve all three of them in parallel, using standard optimization routines. Specifically, we extend the group Lasso penalty (originated in the linear regression literature) in order to impose group-level sparsity on the network's connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We perform an extensive experimental evaluation, by comparing with classical weight decay and Lasso penalties. We show that a sparse version of the group Lasso penalty is able to achieve competitive performances, while at the same time resulting in extremely compact networks with a smaller number of input features. We evaluate both on a toy dataset for handwritten digit recognition, and on multiple realistic large-scale classification problems.