mlsanityData to 7 Oct 2026

Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

arXiv 2105.14368 ↗First posted 29 May 2021stat.ML cs.LG math.STcited by 130

Not on the map. Why?

Abstract (arXiv)

In the past decade the mathematical theory of machine learning has lagged far behind the triumphs of deep neural networks on practical challenges. However, the gap between theory and practice is gradually starting to close. In this paper I will attempt to assemble some pieces of the remarkable and still incomplete mathematical mosaic emerging from the efforts to understand the foundations of deep learning. The two key themes will be interpolation, and its sibling, over-parameterization. Interpolation corresponds to fitting data, even noisy data, exactly. Over-parameterization enables interpolation and provides flexibility to select a right interpolating model. As we will see, just as a physical prism separates colors mixed within a ray of light, the figurative prism of interpolation helps to disentangle generalization and optimization properties within the complex picture of modern Machine Learning. This article is written with belief and hope that clearer understanding of these issues brings us a step closer toward a general theory of deep learning and machine learning.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 57 of its 110 references that are in the map. First posted on arXiv 29 May 2021.

Builds on

57 of 110 references.

References · 57

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  11. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  12. 2019Deep Equilibrium Modelscited by 603
38 more
  1. 2017One pixel attack for fooling deep neural networkscited by 579
  2. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  3. 2014Convex Optimization: Algorithms and Complexitycited by 569
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  9. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2021Deep learning: a statistical viewpointcited by 239
  12. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  13. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  14. 2019Two models of double descent for weak featurescited by 194
  15. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  16. 2018To understand deep learning we need to understand kernel learningcited by 186
  17. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  18. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  19. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  20. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  21. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  22. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  23. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  24. 2019Harmless interpolation of noisy data in regressioncited by 121
  25. 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
  26. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  27. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  28. 2018Does data interpolation contradict statistical optimality?cited by 95
  29. 2020Kernel methods through the roof: handling billions of points efficientlycited by 88
  30. 2016Robustness of classifiers: from adversarial to random noisecited by 87
  31. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  32. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  33. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  34. 2020Neural Kernels Without Tangentscited by 56
  35. 2015Explaining the Success of AdaBoost and Random Forests as Interpolating Classifierscited by 45
  36. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  37. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  38. 2020A Brief Prehistory of Double Descentcited by 44

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 24; 2023: 37; 2024: 25; 2025: 16; 2026: 13.

130 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  3. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
  4. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  5. 2022The Implicit Bias of Benign Overfittingcited by 38
  6. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  7. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  8. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  9. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  10. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  11. 2021Computational Graph Completioncited by 28
  12. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
38 more
  1. 2022A Moment-Matching Approach to Testable Learning and a New Characterization of Rademacher Complexitycited by 27
  2. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  3. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalizationcited by 25
  4. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  5. 2021Foolish Crowds Support Benign Overfittingcited by 24
  6. 2022Convergence of gradient descent for deep neural networkscited by 24
  7. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  8. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  9. 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 20
  10. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  11. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  12. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  13. 2022Benign Underfitting of Stochastic Gradient Descentcited by 18
  14. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  15. 2022Tensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplaycited by 15
  16. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  17. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  18. 2024GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editingcited by 14
  19. 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 13
  20. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  21. 2022Wide and Deep Neural Networks Achieve Optimality for Classificationcited by 12
  22. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  23. 2022Catch-22s of reservoir computingcited by 11
  24. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  25. 2024Coding schemes in neural networks learning classification taskscited by 11
  26. 2024Emergent properties with repeated examplescited by 11
  27. 2025Dense Associative Memory with Epanechnikov Energycited by 10
  28. 2023Why Do Probabilistic Clinical Models Fail To Transport Between Sites?cited by 9
  29. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  30. 2022Generalization for multiclass classification with overparameterized linear modelscited by 8
  31. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  32. 2021The Geometric Occam's Razor Implicit in Deep Learningcited by 7
  33. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  34. 2023Norm-based Generalization Bounds for Compositionally Sparse Neural Networkscited by 7
  35. 2023Efficient kernel surrogates for neural network-based regressioncited by 7
  36. 2021Data splitting improves statistical performance in overparametrized regimescited by 6
  37. 2022Importance Tempering: Group Robustness for Overparameterized Modelscited by 6
  38. 2023Eight challenges in developing theory of intelligencecited by 6

50 most cited of 130 All 130 (JSON)