# A Corrective View of Neural Networks: Representation, Memorization and Learning

arXiv: https://arxiv.org/abs/2002.00274 · HTML: https://mlsanity.com/paper/2002.00274/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2020.

## Built on by (most cited first, 13 of 13)

- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [The staircase property: How hierarchical structure can guide deep learning](https://mlsanity.com/paper/2108.10573/) (2021)
- [Solving Inefficiency of Self-supervised Representation Learning](https://mlsanity.com/paper/2104.08760/) (2021)
- [Impact of Aliasing on Generalization in Deep Convolutional Networks](https://mlsanity.com/paper/2108.03489/) (2021)
- [The Separation Capacity of Random Neural Networks](https://mlsanity.com/paper/2108.00207/) (2021)
- [Scalable Neural Network Kernels](https://mlsanity.com/paper/2310.13225/) (2023)
- [A Recipe for Global Convergence Guarantee in Deep Neural Networks](https://mlsanity.com/paper/2104.05785/) (2021)
- [Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations](https://mlsanity.com/paper/2202.06499/) (2022)
- [EUGens: Efficient, Unified, and General Dense Layers](https://mlsanity.com/paper/2601.22563/) (2026)
- [Efficient uniform approximation using Random Vector Functional Link networks](https://mlsanity.com/paper/2306.17501/) (2023)
- [On the Query Complexity of Training Data Reconstruction in Private Learning](https://mlsanity.com/paper/2303.16372/) (2023)
- [EUGens: Efficient, Unified, and General Dense Layers](https://mlsanity.com/paper/2410.09771/) (2024)
- [WTNN: Weibull-Tailored Neural Networks for survival analysis](https://mlsanity.com/paper/2512.09163/) (2025)

## Abstract (arXiv)

We develop a corrective mechanism for neural network approximation: the total available non-linear units are divided into multiple groups and the first group approximates the function under consideration, the second group approximates the error in approximation produced by the first group and corrects it, the third group approximates the error produced by the first and second groups together and so on. This technique yields several new representation and learning results for neural networks. First, we show that two-layer neural networks in the random features regime (RF) can memorize arbitrary labels for arbitrary points under under Euclidean distance separation condition using tildeO(n) ReLUs which is optimal in n up to logarithmic factors. Next, we give a powerful representation result for two-layer neural networks with ReLUs and smoothed ReLUs which can achieve a squared error of at most ε with O(C(a,d)ε-1/(a+1)) for a ∈ mathbbNcup0 when the function is smooth enough (roughly when it has Theta(ad) bounded derivatives). In certain cases d can be replaced with effective dimension q ll d. Previous results of this type implement Taylor series approximation using deep architectures. We also consider three-layer neural networks and show that the corrective mechanism yields faster representation rates for smooth radial functions. Lastly, we obtain the first O(subpoly(1/ε)) upper bound on the number of neurons required for a two layer network to learn low degree polynomials up to squared error ε via gradient descent. Even though deep networks can express these polynomials with O(polylog(1/ε)) neurons, the best learning bounds on this problem require poly(1/ε) neurons.