# Approximating Continuous Functions by ReLU Nets of Minimal Width

arXiv: https://arxiv.org/abs/1710.11278 · HTML: https://mlsanity.com/paper/1710.11278/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2017.

## Built on by (most cited first, 30 of 122)

- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting](https://mlsanity.com/paper/2201.12886/) (2022)
- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [Towards Lower Bounds on the Depth of ReLU Neural Networks](https://mlsanity.com/paper/2105.14835/) (2021)
- [Your Transformer May Not be as Powerful as You Expect](https://mlsanity.com/paper/2205.13401/) (2022)
- [Transformers are Universal In-context Learners](https://mlsanity.com/paper/2408.01367/) (2024)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopes](https://mlsanity.com/paper/2302.12553/) (2023)
- [Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs](https://mlsanity.com/paper/2110.14416/) (2021)
- [HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees](https://mlsanity.com/paper/2410.10807/) (2024)
- [Achieve the Minimum Width of Neural Networks for Universal Approximation](https://mlsanity.com/paper/2209.11395/) (2022)
- [HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetwork](https://mlsanity.com/paper/2312.15949/) (2023)
- [Machine-Learning Mathematical Structures](https://mlsanity.com/paper/2101.06317/) (2021)
- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [A Discontinuity Capturing Shallow Neural Network for Elliptic Interface Problems](https://mlsanity.com/paper/2106.05587/) (2021)
- [Minimum width for universal approximation using ReLU networks on compact domain](https://mlsanity.com/paper/2309.10402/) (2023)
- [Training Fully Connected Neural Networks is existsℝ-Complete](https://mlsanity.com/paper/2204.01368/) (2022)
- [Activation degree thresholds and expressiveness of polynomial neural networks](https://mlsanity.com/paper/2408.04569/) (2024)
- [Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation](https://mlsanity.com/paper/2305.18460/) (2023)
- [On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks](https://mlsanity.com/paper/2409.00901/) (2024)
- [ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computation](https://mlsanity.com/paper/2102.06635/) (2021)
- [Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks](https://mlsanity.com/paper/2109.11354/) (2021)
- [Minimal Width for Universal Property of Deep RNN](https://mlsanity.com/paper/2211.13866/) (2022)
- [Neural Tangent Kernel Analysis of Deep Narrow Neural Networks](https://mlsanity.com/paper/2202.02981/) (2022)
- [Quantifying Epistemic Uncertainty in Deep Learning](https://mlsanity.com/paper/2110.12122/) (2021)
- [Width is Less Important than Depth in ReLU Neural Networks](https://mlsanity.com/paper/2202.03841/) (2022)
- [Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks](https://mlsanity.com/paper/2306.05674/) (2023)
- [Optimization Over Trained Neural Networks: Taking a Relaxing Walk](https://mlsanity.com/paper/2401.03451/) (2024)
- [PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos](https://mlsanity.com/paper/2404.08921/) (2024)
- [Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networks](https://mlsanity.com/paper/2206.00228/) (2022)

## Abstract (arXiv)

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed din≥ 1, what is the minimal width w so that neural nets with ReLU activations, input dimension din, hidden layer widths at most w, and arbitrary depth can approximate any continuous, real-valued function of din variables arbitrarily well? It turns out that this minimal width is exactly equal to din+1. That is, if all the hidden layer widths are bounded by din, then even in the infinite depth limit, ReLU nets can only express a very limited class of functions, and, on the other hand, any continuous function on the din-dimensional unit cube can be approximated to arbitrary precision by ReLU nets in which all hidden layers have width exactly din+1. Our construction in fact shows that any continuous function f:[0,1]^din→mathbb R^dout can be approximated by a net of width din+dout. We obtain quantitative depth estimates for such an approximation in terms of the modulus of continuity of f.