mlsanityData to 14 Oct 2026

ReLU Deep Neural Networks from the Hierarchical Basis Perspective

arXiv 2105.04156 ↗First posted 10 May 2021math.NA cs.LG cs.NAcited by 12

Not on the map. Why?

Abstract (arXiv)

We study ReLU deep neural networks (DNNs) by investigating their connections with the hierarchical basis method in finite element methods. First, we show that the approximation schemes of ReLU DNNs for x2 and xy are composition versions of the hierarchical basis approximation for these two functions. Based on this fact, we obtain a geometric interpretation and systematic proof for the approximation result of ReLU DNNs for polynomials, which plays an important role in a series of recent exponential approximation results of ReLU DNNs. Through our investigation of connections between ReLU DNNs and the hierarchical basis approximation for x2 and xy, we show that ReLU DNNs with this special structure can be applied only to approximate quadratic functions. Furthermore, we obtain a concise representation to explicitly reproduce any linear finite element function on a two-dimensional uniform mesh by using ReLU DNNs with only two hidden layers.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

16 of 36 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Densely Connected Convolutional Networkscited by 6,720
  3. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  5. 2019DeepXDE: A deep learning library for solving differential equationscited by 619
  6. 2016Error bounds for approximations with deep ReLU networkscited by 574
  7. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  8. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  9. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  10. 2016Benefits of depth in neural networkscited by 250
  11. 2020Deep Network Approximation for Smooth Functionscited by 164
  12. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
4 more
  1. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  2. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  3. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  4. 2020Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularitiescited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 4; 2024: 2; 2025: 3; 2026: 2.

12 papers.

  1. 2021Approximation Properties of Deep ReLU CNNscited by 15
  2. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  3. 2023Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activationcited by 9
  4. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  5. 2023Any Deep ReLU Network is Shallowcited by 6
  6. 2025Some Super-approximation Rates of ReLU Neural Networks for Korobov Functionscited by 3
  7. 2024A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on n-dimensional Classescited by 2
  8. 2025Two-Dimensional Deep ReLU CNN Approximation for Korobov Functions: A Constructive Approachcited by 2
  9. 2024ε-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamicscited by 1
  10. 2025Linear-Size Neural Network Representation of Piecewise Affine Functions in ℝ2cited by 1
  11. 2026Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximationscited by 0
  12. 2026Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergencecited by 0