mlsanityData to 7 Oct 2026

On the symmetries in the dynamics of wide two-layer neural networks

arXiv 2211.08771 ↗First posted 16 Nov 2022cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on the learned parameters and predictors. We first describe a general class of symmetries which, when satisfied by the target function f^* and the input distribution, are preserved by the dynamics. We then study more specific cases. When f^* is odd, we show that the dynamics of the predictor reduces to that of a (non-linearly parameterized) linear predictor, and its exponential convergence can be guaranteed. When f^* has a low-dimensional structure, we prove that the gradient flow PDE reduces to a lower-dimensional PDE. Furthermore, we present informal and numerical arguments that suggest that the input neurons align with the lower-dimensional structure of the problem.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

22 of 37 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  10. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  11. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  12. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
10 more
  1. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  2. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  3. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  4. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  5. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  6. 2019Machine Learning from a Continuous Viewpointcited by 50
  7. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  8. 2020Geometric compression of invariant manifolds in neural netscited by 28
  9. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  10. 2022Polynomial-time Sparse Measure Recovery: From Mean Field Theory to Algorithm Designcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  2. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  3. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  4. 2025Survey on Algorithms for multi-index modelscited by 15
  5. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  6. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  7. 2024Symmetries in Overparametrized Neural Networks: A Mean-Field Viewcited by 2
  8. 2026Singular perturbations and hierarchical learning in two-layer neural networkscited by 0