# An in-depth look at approximation via deep and narrow neural networks

arXiv: https://arxiv.org/abs/2510.07202 · HTML: https://mlsanity.com/paper/2510.07202/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)
- [Towards Lower Bounds on the Depth of ReLU Neural Networks](https://mlsanity.com/paper/2105.14835/) (2021)
- [Collapse of Deep and Narrow Neural Nets](https://mlsanity.com/paper/1808.04947/) (2018)
- [Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depth](https://mlsanity.com/paper/2006.04048/) (2020)
- [Optimal Function Approximation with Relu Neural Networks](https://mlsanity.com/paper/1909.03731/) (2019)
- [Improved weight initialization for deep and narrow feedforward neural network](https://mlsanity.com/paper/2311.03733/) (2023)
- [On decision regions of narrow deep neural networks](https://mlsanity.com/paper/1807.01194/) (2018)
- [Universality of Gradient Descent Neural Network Training](https://mlsanity.com/paper/2007.13664/) (2020)

## Abstract (arXiv)

In 2017, Hanin and Sellke showed that the class of arbitrarily deep, real-valued, feed-forward and ReLU-activated networks of width w forms a dense subset of the space of continuous functions on R^n, with respect to the topology of uniform convergence on compact sets, if and only if w>n holds. To show the necessity, a concrete counterexample function f:R^n->R was used. In this note we actually approximate this very f by neural networks in the two cases w=n and w=n+1 around the aforementioned threshold. We study how the approximation quality behaves if we vary the depth and what effect (spoiler alert: dying neurons) cause that behavior.