An in-depth look at approximation via deep and narrow neural networks
Not on the map. Why?
Abstract (arXiv)
In 2017, Hanin and Sellke showed that the class of arbitrarily deep, real-valued, feed-forward and ReLU-activated networks of width w forms a dense subset of the space of continuous functions on R^n, with respect to the topology of uniform convergence on compact sets, if and only if w>n holds. To show the necessity, a concrete counterexample function f:R^n->R was used. In this note we actually approximate this very f by neural networks in the two cases w=n and w=n+1 around the aforementioned threshold. We study how the approximation quality behaves if we vary the depth and what effect (spoiler alert: dying neurons) cause that behavior.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 23 references that are in the map. First posted on arXiv 8 Oct 2025.
Builds on
10 of 23 references.
References · 10
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 120
- 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
- 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
- 2018Collapse of Deep and Narrow Neural Netscited by 37
- 2020Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depthcited by 13
- 2019Optimal Function Approximation with Relu Neural Networkscited by 9
- 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
- 2018On decision regions of narrow deep neural networkscited by 5
- 2020Universality of Gradient Descent Neural Network Trainingcited by 1
Built on by
Not cited yet.