mlsanityData to 7 Oct 2026

Learning in High Dimension Always Amounts to Extrapolation

arXiv 2110.09485 ↗First posted 18 Oct 2021cs.LG cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample x whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when x falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional (>100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 4 of its 28 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

4 of 28 references.

References · 4

  1. 2018To understand deep learning we need to understand kernel learningcited by 186
  2. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  3. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  4. 2020When does gradient descent with logistic loss find interpolating two-layer networks?cited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 26; 2023: 13; 2024: 8; 2025: 6; 2026: 5.

62 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  4. 2022Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practicescited by 54
  5. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  6. 2022A Survey of Robust Adversarial Training in Pattern Recognition: Fundamental, Theory, and Methodologiescited by 44
  7. 2021Mixed-Integer Optimization with Constraint Learningcited by 30
  8. 2022On the Generalization Mystery in Deep Learningcited by 25
  9. 2022Out-Of-Distribution Detection Is Not All You Needcited by 23
  10. 2023Deep Generative Models for Detector Signature Simulation: A Taxonomic Reviewcited by 18
  11. 2023Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirscited by 15
  12. 2023Interpreting the Curse of Dimensionality from Distance Concentration and Manifold Effectcited by 14
38 more
  1. 2025Numerical models outperform AI weather forecasts of record-breaking extremescited by 14
  2. 2023Improved uncertainty quantification for neural networks with Bayesian last layercited by 11
  3. 2023Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifoldscited by 9
  4. 2023Neural oscillators for generalization of physics-informed machine learningcited by 9
  5. 2022Counterfactual Explanations Using Optimization With Constraint Learningcited by 8
  6. 2023Asymmetric Certified Robustness via Feature-Convex Neural Networkscited by 8
  7. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  8. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  9. 2024How language models extrapolate outside the training data: A case study in Textualized Gridworldcited by 7
  10. 2021Adaptive Feature Interpolation for Low-Shot Image Generationcited by 6
  11. 2022Decision boundaries and convex hulls in the feature space that deep learning functions learn from imagescited by 5
  12. 2021Hierarchical transfer learning with applications for electricity load forecastingcited by 4
  13. 2022A Precis of Language Models are not Models of Languagecited by 4
  14. 2024Towards certification: A complete statistical validation pipeline for supervised learning in industrycited by 4
  15. 2022Less is More: Rethinking Few-Shot Learning and Recurrent Neural Netscited by 3
  16. 2022Logical Tasks for Measuring Extrapolation and Rule Comprehensioncited by 3
  17. 2022A survey of deep learning optimizers -- first and second order methodscited by 3
  18. 2023Training Dynamics of Deep Network Linear Regionscited by 3
  19. 2024On Generalization for Generative Flow Networkscited by 3
  20. 2025A Geometric Unification of Concept Learning with Concept Conescited by 3
  21. 2021Manifold embedding data-driven mechanicscited by 2
  22. 2022Manifoldron: Direct Space Partition via Manifold Discoverycited by 2
  23. 2022Manifold Characteristics That Predict Downstream Task Performancecited by 2
  24. 2022Membership Inference Attacks via Adversarial Examplescited by 2
  25. 2022Towards an AI-based Early Warning System for Bridge Scourcited by 2
  26. 2023Pathway toward prior knowledge-integrated machine learning in engineeringcited by 2
  27. 2023Using causal inference to avoid fallouts in data-driven parametric analysis: a case study in the architecture, engineering, and construction industrycited by 2
  28. 2022To what extent should we trust AI models when they extrapolate?cited by 1
  29. 2022Emulating Quantum Dynamics with Neural Networks via Knowledge Distillationcited by 1
  30. 2022Over-parameterization: A Necessary Condition for Models that Extrapolatecited by 1
  31. 2022Split personalities in Bayesian Neural Networks: the case for full marginalisationcited by 1
  32. 2022Learning inducing points and uncertainty on molecular data by scalable variational Gaussian processescited by 1
  33. 2023Topological Parallax: A Geometric Specification for Deep Perception Modelscited by 1
  34. 2023Deep Neural Networks Tend To Extrapolate Predictablycited by 1
  35. 2024GROOT: Effective Design of Biological Sequences with Limited Experimental Datacited by 1
  36. 2025Network Embedding Exploration Tool (NEExT)cited by 1
  37. 2025Flexible Swarm Learning May Outpace Foundation Models in Essential Taskscited by 1
  38. 2025A Critical Assessment of PINNs and Operator Learning for Geotechnical Engineeringcited by 1

50 most cited of 62 All 62 (JSON)