mlsanityData to 7 Oct 2026

Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsity

arXiv 2205.15809 ↗First posted 31 May 2022stat.ML cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

We study the loss surface of DNNs with L2 regularization. We show that the loss in terms of the parameters can be reformulated into a loss in terms of the layerwise activations Zℓ of the training set. This reformulation reveals the dynamics behind feature learning: each hidden representations Zℓ are optimal w.r.t. to an attraction/repulsion problem and interpolate between the input and output representations, keeping as little information from the input as necessary to construct the activation of the next layer. For positively homogeneous non-linearities, the loss can be further reformulated in terms of the covariances of the hidden representations, which takes the form of a partially convex optimization over a convex cone. This second reformulation allows us to prove a sparsity result for homogeneous DNNs: any local minimum of the L2-regularized loss can be achieved with at most N(N+1) neurons in each hidden layer (where N is the size of the training set). We show that this bound is tight by giving an example of a local minimum that requires N2/4 hidden neurons. But we also observe numerically that in more traditional settings much less than N2 neurons are required to reach the minima.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

20 of 32 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  10. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  11. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  12. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
8 more
  1. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  2. 2019How do infinite width bounded norm networks look in function space?cited by 105
  3. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  4. 2015Towards Biologically Plausible Deep Learningcited by 66
  5. 2019Biologically plausible deep learning -- but how far can we go with shallow networks?cited by 25
  6. 2017Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?cited by 20
  7. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
  8. 2019Structured and Deep Similarity Matching via Structured and Deep Hebbian Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 3; 2025: 4; 2026: 2.

17 papers.

Sort:

By area:

  1. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  2. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  3. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  4. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  5. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 17
  6. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  7. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  8. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  9. 2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
  10. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  11. 2022PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networkscited by 2
  12. 2025On the Cone Effect in the Learning Dynamicscited by 2
5 more
  1. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  2. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  3. 2025Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networkscited by 0
  4. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  5. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0