Long-Tailed Learning Requires Feature Learning
Not on the map. Why?
Abstract (arXiv)
We propose a simple data model inspired from natural data such as text or images, and use it to study the importance of learning features in order to achieve good generalization. Our data model follows a long-tailed distribution in the sense that some rare subcategories have few representatives in the training set. In this context we provide evidence that a learner succeeds if and only if it identifies the correct features, and moreover derive non-asymptotic generalization error bounds that precisely quantify the penalty that one must pay for not learning features.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 29 May 2022.
Builds on
21 of 28 references.
References · 21
- 2016Layer Normalizationcited by 5,669
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
- 2017Spectrally-normalized margin bounds for neural networkscited by 686
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
- 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
- 2017Size-Independent Sample Complexity of Neural Networkscited by 290
- 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
9 more
- 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
- 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 104
- 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
- 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
- 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
Built on by
Citing papers per year: 2025: 1.
1 paper.