mlsanityData to 7 Oct 2026

Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective

arXiv 2403.14917 ↗First posted 22 Mar 2024cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the feature learning ability of two-layer neural networks in the mean-field regime through the lens of kernel methods. To focus on the dynamics of the kernel induced by the first layer, we utilize a two-timescale limit, where the second layer moves much faster than the first layer. In this limit, the learning problem is reduced to the minimization problem over the intrinsic kernel. Then, we show the global convergence of the mean-field Langevin dynamics and derive time and particle discretization error. We also demonstrate that two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than any kernel methods, and neural networks acquire data-dependent kernel which aligns with the target function. In addition, we develop a label noise procedure, which converges to the global optimum and show that the degrees of freedom appears as an implicit regularization.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

31 of 50 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  6. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  7. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  8. 2016A vector-contraction inequality for Rademacher complexitiescited by 141
  9. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  10. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  11. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
19 more
  1. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  2. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  3. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  4. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  5. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  6. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
  7. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  8. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  9. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  10. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  11. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  12. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  13. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  14. 2019On the minimax optimality and superiority of deep neural network learning over sparse parameter spacescited by 24
  15. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  16. 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
  17. 2023Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reductioncited by 19
  18. 2023Leveraging the two timescale regime to demonstrate convergence of neural networkscited by 18
  19. 2019Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representationscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 4.

12 papers.

  1. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  2. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 14
  3. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  4. 2025Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regimecited by 4
  5. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  6. 2025Mirror Mean-Field Langevin Dynamicscited by 3
  7. 2026Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networkscited by 1
  8. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  9. 2025Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attentioncited by 0
  10. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  11. 2026Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Ratecited by 0
  12. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0