# Learning in High Dimension Always Amounts to Extrapolation

arXiv: https://arxiv.org/abs/2110.09485 · HTML: https://mlsanity.com/paper/2110.09485/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 28 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 4 of 4)

- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization](https://mlsanity.com/paper/2008.06786/) (2020)
- [When does gradient descent with logistic loss find interpolating two-layer networks?](https://mlsanity.com/paper/2012.02409/) (2020)

## Built on by (most cited first, 30 of 66)

- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)
- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention](https://mlsanity.com/paper/2202.05798/) (2022)
- [A Survey of Robust Adversarial Training in Pattern Recognition: Fundamental, Theory, and Methodologies](https://mlsanity.com/paper/2203.14046/) (2022)
- [Mixed-Integer Optimization with Constraint Learning](https://mlsanity.com/paper/2111.04469/) (2021)
- [On the Generalization Mystery in Deep Learning](https://mlsanity.com/paper/2203.10036/) (2022)
- [Out-Of-Distribution Detection Is Not All You Need](https://mlsanity.com/paper/2211.16158/) (2022)
- [Deep Generative Models for Detector Signature Simulation: A Taxonomic Review](https://mlsanity.com/paper/2312.09597/) (2023)
- [Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirs](https://mlsanity.com/paper/2307.12904/) (2023)
- [Interpreting the Curse of Dimensionality from Distance Concentration and Manifold Effect](https://mlsanity.com/paper/2401.00422/) (2023)
- [Numerical models outperform AI weather forecasts of record-breaking extremes](https://mlsanity.com/paper/2508.15724/) (2025)
- [Improved uncertainty quantification for neural networks with Bayesian last layer](https://mlsanity.com/paper/2302.10975/) (2023)
- [Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds](https://mlsanity.com/paper/2303.12307/) (2023)
- [Neural oscillators for generalization of physics-informed machine learning](https://mlsanity.com/paper/2308.08989/) (2023)
- [Counterfactual Explanations Using Optimization With Constraint Learning](https://mlsanity.com/paper/2209.10997/) (2022)
- [Asymmetric Certified Robustness via Feature-Convex Neural Networks](https://mlsanity.com/paper/2302.01961/) (2023)
- [Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study](https://mlsanity.com/paper/2209.07736/) (2022)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [How language models extrapolate outside the training data: A case study in Textualized Gridworld](https://mlsanity.com/paper/2406.15275/) (2024)
- [Adaptive Feature Interpolation for Low-Shot Image Generation](https://mlsanity.com/paper/2112.02450/) (2021)
- [Decision boundaries and convex hulls in the feature space that deep learning functions learn from images](https://mlsanity.com/paper/2202.04052/) (2022)
- [Hierarchical transfer learning with applications for electricity load forecasting](https://mlsanity.com/paper/2111.08512/) (2021)
- [A Precis of Language Models are not Models of Language](https://mlsanity.com/paper/2205.07634/) (2022)
- [Towards certification: A complete statistical validation pipeline for supervised learning in industry](https://mlsanity.com/paper/2411.02075/) (2024)
- [Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets](https://mlsanity.com/paper/2209.14267/) (2022)
- [Logical Tasks for Measuring Extrapolation and Rule Comprehension](https://mlsanity.com/paper/2211.07727/) (2022)
- [A survey of deep learning optimizers -- first and second order methods](https://mlsanity.com/paper/2211.15596/) (2022)
- [Training Dynamics of Deep Network Linear Regions](https://mlsanity.com/paper/2310.12977/) (2023)

## Abstract (arXiv)

The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample x whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when x falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional (>100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.