mlsanityData to 14 Oct 2026

Deep Learning Meets Sparse Regularization: A Signal Processing Perspective

arXiv 2301.09554 ↗First posted 23 Jan 2023stat.ML cs.LG eess.SPcited by 21

Not on the map. Why?

Abstract (arXiv)

Deep learning has been wildly successful in practice and most state-of-the-art machine learning methods are based on neural networks. Lacking, however, is a rigorous mathematical theory that adequately explains the amazing performance of deep neural networks. In this article, we present a relatively new mathematical framework that provides the beginning of a deeper understanding of deep learning. This framework precisely characterizes the functional properties of neural networks that are trained to fit to data. The key mathematical tools which support this framework include transform-domain sparse regularization, the Radon transform of computed tomography, and approximation theory, which are all techniques deeply rooted in signal processing. This framework explains the effect of weight decay regularization in neural network training, the use of skip connections and low-rank weight matrices in network architectures, the role of sparsity in neural networks, and explains why neural networks can perform well in high-dimensional problems.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 22 of its 55 references that are in the map. First posted on arXiv 23 Jan 2023.

Builds on

22 of 55 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2013Do Deep Nets Really Need to be Deep?cited by 474
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  7. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  8. 2019How do infinite width bounded norm networks look in function space?cited by 106
  9. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  10. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  11. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  12. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
10 more
  1. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  2. 2018Mad Max: Affine Spline Insights into Deep Learningcited by 55
  3. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  4. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  5. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  6. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  7. 2020Are wider nets better given the same number of parameters?cited by 27
  8. 2022Ridges, Neural Networks, and the Radon Transformcited by 18
  9. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  10. 2022From Kernel Methods to Neural Networks: A Unifying Variational Formulationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 4; 2026: 4.

21 papers.

Sort:

  1. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  2. 2024Applied Causal Inference Powered by ML and AIcited by 30
  3. 2023Penalising the biases in norm regularisation enforces sparsitycited by 19
  4. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 18
  5. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  6. 2024Stereographic Spherical Sliced Wasserstein Distancescited by 12
  7. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 9
  8. 2023Weighted variation spaces and approximation by shallow ReLU networkscited by 7
  9. 2023Function-Space Optimality of Neural Architectures with Multivariate Nonlinearitiescited by 5
  10. 2024ReLUs Are Sufficient for Learning Implicit Neural Representationscited by 5
  11. 2024Random ReLU Neural Networks as Non-Gaussian Processescited by 3
  12. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 3
9 more
  1. 2024Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Modelscited by 2
  2. 2025A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysiscited by 2
  3. 2025Nonasymptotic Convergence Rates for Plug-and-Play Methods With MMSE Denoiserscited by 2
  4. 2024Towards a Sampling Theory for Implicit Neural Representationscited by 1
  5. 2026Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspectivecited by 1
  6. 2024Accelerating Inference of Networks in the Frequency Domaincited by 0
  7. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0
  8. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0
  9. 2026Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layerscited by 0