mlsanityData to 7 Oct 2026

Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics

arXiv 2408.07254 ↗First posted 14 Aug 2024stat.ML cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

We study the problem of learning multi-index models in high-dimensions using a two-layer neural network trained with the mean-field Langevin algorithm. Under mild distributional assumptions on the data, we characterize the effective dimension deff that controls both sample and computational complexity by utilizing the adaptivity of neural networks to latent low-dimensional structures. When the data exhibit such a structure, deff can be significantly smaller than the ambient dimension. We prove that the sample complexity grows almost linearly with deff, bypassing the limitations of the information and generative exponents that appeared in recent analyses of gradient-based feature learning. On the other hand, the computational complexity may inevitably grow exponentially with deff in the worst-case scenario. Motivated by improving computational complexity, we take the first steps towards polynomial time convergence of the mean-field Langevin algorithm by investigating a setting where the weights are constrained to be on a compact manifold with positive Ricci curvature, such as the hypersphere. There, we study assumptions under which polynomial time convergence is achievable, whereas similar assumptions in the Euclidean setting lead to exponential time complexity.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 14 Aug 2024.

Builds on

45 of 73 references.

References · 45

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  10. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  11. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  12. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
33 more
  1. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  2. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  3. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  4. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  5. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  6. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
  7. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
  8. 2023Learning time-scales in two-layers neural networkscited by 57
  9. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  10. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  11. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  12. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  13. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  14. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  15. 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
  16. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
  17. 2020Learning Polynomials of Few Relevant Dimensionscited by 34
  18. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  19. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  20. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  21. 2020Riemannian Langevin Algorithm for Solving Semidefinite Programscited by 22
  22. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  23. 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
  24. 2021Convergence Rates of Gradient Methods for Convex Optimization in the Space of Measurescited by 19
  25. 2023Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reductioncited by 19
  26. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  27. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  28. 2024On the Complexity of Learning Sparse Functions with Statistical and Gradient Queriescited by 13
  29. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  30. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  31. 2024Sampling from the Mean-Field Stationary Distributioncited by 7
  32. 2024Mean-Field Langevin Dynamics for Signed Measures via a Bilevel Approachcited by 6
  33. 2024Pruning is Optimal for Learning Sparse Features in High-Dimensionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 2.

14 papers.

Sort:

  1. 2025Survey on Algorithms for multi-index modelscited by 15
  2. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 6
  3. 2024Robust Feature Learning for Multi-Index Models in High Dimensionscited by 5
  4. 2025Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemblecited by 5
  5. 2025Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descentcited by 4
  6. 2025Mirror Mean-Field Langevin Dynamicscited by 3
  7. 2025Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphscited by 3
  8. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2
  9. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  10. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  11. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  12. 2025Provably Extracting the Features from a General Superpositioncited by 1
2 more
  1. 2026Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networkscited by 1
  2. 2026Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networkscited by 0