# ReLU Deep Neural Networks from the Hierarchical Basis Perspective

arXiv: https://arxiv.org/abs/2105.04156 · HTML: https://mlsanity.com/paper/2105.04156/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions](https://mlsanity.com/paper/1807.00297/) (2018)
- [Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities](https://mlsanity.com/paper/2010.12217/) (2020)

## Built on by (most cited first, 12 of 12)

- [Approximation Properties of Deep ReLU CNNs](https://mlsanity.com/paper/2109.00190/) (2021)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activation](https://mlsanity.com/paper/2312.16483/) (2023)
- [Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions](https://mlsanity.com/paper/2312.14276/) (2023)
- [Any Deep ReLU Network is Shallow](https://mlsanity.com/paper/2306.11827/) (2023)
- [Some Super-approximation Rates of ReLU Neural Networks for Korobov Functions](https://mlsanity.com/paper/2507.10345/) (2025)
- [A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on n-dimensional Classes](https://mlsanity.com/paper/2404.06104/) (2024)
- [Two-Dimensional Deep ReLU CNN Approximation for Korobov Functions: A Constructive Approach](https://mlsanity.com/paper/2503.07976/) (2025)
- [ε-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics](https://mlsanity.com/paper/2412.05144/) (2024)
- [Linear-Size Neural Network Representation of Piecewise Affine Functions in ℝ2](https://mlsanity.com/paper/2503.13001/) (2025)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)
- [Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence](https://mlsanity.com/paper/2609.25874/) (2026)

## Abstract (arXiv)

We study ReLU deep neural networks (DNNs) by investigating their connections with the hierarchical basis method in finite element methods. First, we show that the approximation schemes of ReLU DNNs for x2 and xy are composition versions of the hierarchical basis approximation for these two functions. Based on this fact, we obtain a geometric interpretation and systematic proof for the approximation result of ReLU DNNs for polynomials, which plays an important role in a series of recent exponential approximation results of ReLU DNNs. Through our investigation of connections between ReLU DNNs and the hierarchical basis approximation for x2 and xy, we show that ReLU DNNs with this special structure can be applied only to approximate quadratic functions. Furthermore, we obtain a concise representation to explicitly reproduce any linear finite element function on a two-dimensional uniform mesh by using ReLU DNNs with only two hidden layers.