mlsanityData to 14 Oct 2026

Random Feature Amplification: Feature Learning and Generalization in Neural Networks

arXiv 2202.07626 ↗First posted 15 Feb 2022cs.LG math.ST stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

In this work, we provide a characterization of the feature-learning process in two-layer ReLU networks trained by gradient descent on the logistic loss following random initialization. We consider data with binary labels that are generated by an XOR-like function of the input features. We permit a constant fraction of the training labels to be corrupted by an adversary. We show that, although linear classifiers are no better than random guessing for the distribution we consider, two-layer ReLU networks trained by gradient descent achieve generalization error close to the label noise rate. We develop a novel proof technique that shows that at initialization, the vast majority of neurons function as random features that are only weakly correlated with useful features, and the gradient descent dynamics 'amplify' these weak, random features to strong, useful features.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

37 of 57 references.

References · 37

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2019Benign Overfitting in Linear Regressioncited by 549
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  9. 2016Benefits of depth in neural networkscited by 250
  10. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  11. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  12. 2015The Power of Depth for Feedforward Neural Networkscited by 183
25 more
  1. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  2. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  3. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  4. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  5. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  6. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  7. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  8. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  9. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  10. 2020Learning Parities with Neural Networkscited by 78
  11. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  12. 2020Learning a Single Neuron with Gradient Methodscited by 50
  13. 2017Depth Separation for Neural Networkscited by 46
  14. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  15. 2020Learning Halfspaces with Massart Noise Under Structured Distributionscited by 45
  16. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  17. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  18. 2021Early-stopped neural networks are consistentcited by 34
  19. 2021On the Power of Differentiable Learning versus PAC and SQ Learningcited by 28
  20. 2019Depth Separations in Neural Networks: What is Actually Being Separated?cited by 27
  21. 2021Properties of the After Kernelcited by 27
  22. 2020Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexitycited by 25
  23. 2019Is Deeper Better only when Shallow is Good?cited by 24
  24. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  25. 2021Optimization-Based Separations for Neural Networkscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 11; 2024: 5; 2025: 5; 2026: 8.

35 papers.

Sort:

  1. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  2. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  3. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  4. 2023On the Role of Attention in Prompt-tuningcited by 50
  5. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  6. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  7. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 39
  8. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  9. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  10. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  11. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  12. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
23 more
  1. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14
  2. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  3. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  4. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  5. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  6. 2022Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergencecited by 6
  7. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  8. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  9. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 4
  10. 2026From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretabilitycited by 4
  11. 2023Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learningcited by 3
  12. 2025Transformers Provably Learn Sparse XOR with Polylogarithmic Parameterscited by 3
  13. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  14. 2025Online Federation For Mixtures of Proprietary Agents with Black-Box Encoderscited by 1
  15. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  16. 2026Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networkscited by 1
  17. 2023Do Neural Networks Generalize from Self-Averaging Sub-classifiers in the Same Way As Adaptive Boosting?cited by 0
  18. 2025Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernelcited by 0
  19. 2026Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Datacited by 0
  20. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0
  21. 2026Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descentcited by 0
  22. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0
  23. 2026Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learningcited by 0