mlsanityData to 7 Oct 2026

A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Features

arXiv 2206.01717 ↗First posted 3 Jun 2022cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

An important characteristic of neural networks is their ability to learn representations of the input data with effective features for prediction, which is believed to be a key factor to their superior empirical performance. To better understand the source and benefit of feature learning in neural networks, we consider learning problems motivated by practical data, where the labels are determined by a set of class relevant patterns and the inputs are generated from these along with some background patterns. We prove that neural networks trained by gradient descent can succeed on these problems. The success relies on the emergence and improvement of effective features, which are learned among exponentially many candidates efficiently by exploiting the data (in particular, the structure of the input distribution). In contrast, no linear models on data-independent features of polynomial sizes can learn to as good errors. Furthermore, if the specific input structure is removed, then no polynomial algorithm in the Statistical Query model can learn even weakly. These results provide theoretical evidence showing that feature learning in neural networks depends strongly on the input structure and leads to the superior performance. Our preliminary experimental results on synthetic and real data also provide positive support.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 19 of its 58 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

19 of 58 references.

References · 19

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  6. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  7. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  8. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  9. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  10. 2019Are All Layers Created Equal?cited by 94
  11. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  12. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
7 more
  1. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  2. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  3. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  4. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  5. 2020Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexitycited by 25
  6. 2020Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learningcited by 18
  7. 2019Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representationscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 18; 2025: 9; 2026: 5.

49 papers.

Sort:

By area:

  1. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  2. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  3. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  4. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  5. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  6. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  7. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  8. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  9. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  10. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  11. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  12. 2024Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspectivecited by 24
37 more
  1. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  2. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  3. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  4. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  5. 2023On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Explorationcited by 17
  6. 2023Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networkscited by 16
  7. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  8. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 15
  9. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  10. 2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
  11. 2024What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encodingcited by 14
  12. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  13. 2022Feature selection with gradient descent on two-layer networks in low-rotation regimescited by 13
  14. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  15. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  16. 2025Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometrycited by 9
  17. 2024Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysiscited by 8
  18. 2024How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalancecited by 6
  19. 2023Structured and Fast Optimization: The Kronecker SGD Algorithmcited by 5
  20. 2024SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learningcited by 5
  21. 2025Approximating Latent Manifolds in Neural Networks via Vanishing Idealscited by 5
  22. 2024How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networkscited by 4
  23. 2023Pruning Before Training May Improve Generalization, Provablycited by 3
  24. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  25. 2026Explaining Grokking in Transformers through the Lens of Inductive Biascited by 2
  26. 2023On Feature Scaling of Recursive Feature Machinescited by 1
  27. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  28. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  29. 2025Unifying Learning Dynamics and Generalization in Transformers Scaling Lawcited by 1
  30. 2026Visual prompting reimagined: The power of the Activation Promptscited by 1
  31. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  32. 2024Half-Space Feature Learning in Neural Networkscited by 0
  33. 2025Dictionary Learning: The Complexity of Learning Sparse Superposed Features with Feedbackcited by 0
  34. 2025Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regimecited by 0
  35. 2025A simple mean field model of feature learningcited by 0
  36. 2026Neuron Block Dynamics for XOR Classification with Zero-Margincited by 0
  37. 2026Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guaranteescited by 0