mlsanityData to 14 Oct 2026

Sliding down the stairs: how correlated latent variables accelerate learning with neural networks

arXiv 2404.08602 ↗First posted 12 Apr 2024stat.ML cond-mat.stat-mech cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Neural networks extract features from data using stochastic gradient descent (SGD). In particular, higher-order input cumulants (HOCs) are crucial for their performance. However, extracting information from the pth cumulant of d-dimensional inputs is computationally hard: the number of samples required to recover a single direction from an order-p tensor (tensor PCA) using online SGD grows as dp-1, which is prohibitive for high-dimensional inputs. This result raises the question of how neural networks extract relevant directions from the HOCs of their inputs efficiently. Here, we show that correlations between latent variables along the directions encoded in different input cumulants speed up learning from higher-order correlations. We show this effect analytically by deriving nearly sharp thresholds for the number of samples required by a single neuron to weakly-recover these directions using online SGD from a random start in high dimensions. Our analytical results are confirmed in simulations of two-layer neural networks and unveil a new mechanism for hierarchical learning in neural networks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

42 of 60 references.

References · 42

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  5. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  6. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  7. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  8. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  11. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  12. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
30 more
  1. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  2. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  3. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  4. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  5. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  6. 2019Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratiocited by 85
  7. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  8. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  9. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  10. 2014A statistical model for tensor PCAcited by 72
  11. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  12. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  13. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  14. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  15. 2023Learning time-scales in two-layers neural networkscited by 57
  16. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  17. 2017The power of sum-of-squares for detecting hidden structurescited by 54
  18. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  19. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  20. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  21. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  22. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 31
  23. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  24. 2017Bayesian estimation from few samples: community detection and related problemscited by 28
  25. 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
  26. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 26
  27. 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
  28. 2018Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inferencecited by 24
  29. 2022Decomposing neural networks as mappings of correlation functionscited by 12
  30. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 5.

12 papers.

  1. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  2. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 24
  3. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  4. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  5. 2026A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalizationcited by 8
  6. 2025Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensionscited by 7
  7. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  8. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  9. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  10. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  11. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  12. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0