# Layer-wise Geometric Approximation Rates for Deep Networks

arXiv: https://arxiv.org/abs/2604.20219 · HTML: https://mlsanity.com/paper/2604.20219/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 55 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Understanding Gradient Descent on Edge of Stability in Deep Learning](https://mlsanity.com/paper/2205.09745/) (2022)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Neural Network Approximation: Three Hidden Layers Are Enough](https://mlsanity.com/paper/2010.14075/) (2020)
- [Theory of the Frequency Principle for General Deep Neural Networks](https://mlsanity.com/paper/1906.09235/) (2019)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Nonlinear Approximation via Compositions](https://mlsanity.com/paper/1902.10170/) (2019)
- [Elementary superexpressive activations](https://mlsanity.com/paper/2102.10911/) (2021)
- [A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems](https://mlsanity.com/paper/1909.11759/) (2019)
- [Deep Network Approximation: Beyond ReLU to Diverse Activation Functions](https://mlsanity.com/paper/2307.06555/) (2023)
- [On a Sparse Shortcut Topology of Artificial Neural Networks](https://mlsanity.com/paper/1811.09003/) (2018)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [Interpolation, Approximation and Controllability of Deep Neural Networks](https://mlsanity.com/paper/2309.06015/) (2023)
- [Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning](https://mlsanity.com/paper/2410.16105/) (2024)
- [Approximation in shift-invariant spaces with deep ReLU neural networks](https://mlsanity.com/paper/2005.11949/) (2020)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)

## Built on by (most cited first, 2 of 2)

- [Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations](https://mlsanity.com/paper/2608.31157/) (2026)
- [Staged Depth Training: A Representation Curriculum for PINNs](https://mlsanity.com/paper/2609.30299/) (2026)

## Abstract (arXiv)

Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width 2dN+d+2 and any prescribed finite depth such that each intermediate readout Phiℓ is itself an approximant to the target function f. For f∈ Lp([0,1]d) with p∈ [1,∞), the approximation error of Phiℓ is controlled by (2d+1) times the Lp modulus of continuity at the geometric scale N-ℓ for all ℓ. The estimate reduces to the geometric rate (2d+1)N-ℓ if f is 1-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism. For every prescribed terminal depth, the construction yields a finite nested family of prefix readouts whose earlier correction terms remain embedded in later readouts. Thus the approximation may be truncated within the prescribed depth range once the desired certified accuracy is reached.