# Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depth

arXiv: https://arxiv.org/abs/2006.04048 · HTML: https://mlsanity.com/paper/2006.04048/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 13 of 13)

- [The staircase property: How hierarchical structure can guide deep learning](https://mlsanity.com/paper/2108.10573/) (2021)
- [Theoretical limitations of multi-layer Transformer](https://mlsanity.com/paper/2412.02975/) (2024)
- [On the Optimal Memorization Power of ReLU Neural Networks](https://mlsanity.com/paper/2110.03187/) (2021)
- [On the Representation of Solutions to Elliptic PDEs in Barron Spaces](https://mlsanity.com/paper/2106.07539/) (2021)
- [Depth separation beyond radial functions](https://mlsanity.com/paper/2102.01621/) (2021)
- [Width is Less Important than Depth in ReLU Neural Networks](https://mlsanity.com/paper/2202.03841/) (2022)
- [Size and Depth Separation in Approximating Benign Functions with Neural Networks](https://mlsanity.com/paper/2102.00314/) (2021)
- [Approximation results for Gradient Descent trained Shallow Neural Networks in 1d](https://mlsanity.com/paper/2209.08399/) (2022)
- [Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem](https://mlsanity.com/paper/2110.10295/) (2021)
- [Approximation Results for Gradient Descent trained Neural Networks](https://mlsanity.com/paper/2309.04860/) (2023)
- [Approximation and Gradient Descent Training with Neural Networks](https://mlsanity.com/paper/2405.11696/) (2024)
- [Approximation Rates of Shallow Neural Networks: Barron Spaces, Activation Functions and Optimality Analysis](https://mlsanity.com/paper/2510.18388/) (2025)
- [An in-depth look at approximation via deep and narrow neural networks](https://mlsanity.com/paper/2510.07202/) (2025)

## Abstract (arXiv)

We prove sharp dimension-free representation results for neural networks with D ReLU layers under square loss for a class of functions GD defined in the paper. These results capture the precise benefits of depth in the following sense: 1. The rates for representing the class of functions GD via D ReLU layers is sharp up to constants, as shown by matching lower bounds. 2. For each D, GD subseteq GD+1 and as D grows the class of functions GD contains progressively less smooth functions. 3. If Dprime < D, then the approximation rate for the class GD achieved by depth Dprime networks is strictly worse than that achieved by depth D networks. This constitutes a fine-grained characterization of the representation power of feedforward networks of arbitrary depth D and number of neurons N, in contrast to existing representation results which either require D growing quickly with N or assume that the function being represented is highly smooth. In the latter case similar rates can be obtained with a single nonlinear layer. Our results confirm the prevailing hypothesis that deeper networks are better at representing less smooth functions, and indeed, the main technical novelty is to fully exploit the fact that deep networks can produce highly oscillatory functions with few activation functions.