# Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels

arXiv: https://arxiv.org/abs/2005.10807 · HTML: https://mlsanity.com/paper/2005.10807/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 19 of 19)

- [Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks](https://mlsanity.com/paper/2205.02902/) (2022)
- [Duality for Neural Networks through Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2211.05020/) (2022)
- [An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximation](https://mlsanity.com/paper/2104.07794/) (2021)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [A Quantitative Functional Central Limit Theorem for Shallow Neural Networks](https://mlsanity.com/paper/2306.16932/) (2023)
- [Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks](https://mlsanity.com/paper/2404.18769/) (2024)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms](https://mlsanity.com/paper/2508.05141/) (2025)
- [A Class of Dimension-free Metrics for the Convergence of Empirical Measures](https://mlsanity.com/paper/2104.12036/) (2021)
- [Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality](https://mlsanity.com/paper/2209.01173/) (2022)
- [Reinforcement Learning with Function Approximation: From Linear to Nonlinear](https://mlsanity.com/paper/2302.09703/) (2023)
- [Embeddings between Barron spaces with higher order activation functions](https://mlsanity.com/paper/2305.15839/) (2023)
- [Minimum norm interpolation by perceptra: Explicit regularization and implicit bias](https://mlsanity.com/paper/2311.06138/) (2023)
- [Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activation](https://mlsanity.com/paper/2203.13410/) (2022)
- [Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimates](https://mlsanity.com/paper/2303.08587/) (2023)
- [Curse of Dimensionality in Neural Network Optimization](https://mlsanity.com/paper/2502.05360/) (2025)
- [Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Method](https://mlsanity.com/paper/2607.25100/) (2026)
- [Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks](https://mlsanity.com/paper/2103.16355/) (2021)

## Abstract (arXiv)

We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor L2-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the L2-topology.