mlsanityData to 7 Oct 2026

On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network

arXiv 2301.12353 ↗First posted 29 Jan 2023cs.LG math.DS stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper explores the expressive power of deep neural networks through the framework of function compositions. We demonstrate that the repeated compositions of a single fixed-size ReLU network exhibit surprising expressive power, despite the limited expressive capabilities of the individual network itself. Specifically, we prove by construction that L2circ boldsymbolgcirc rcirc boldsymbolL1 can approximate 1-Lipschitz continuous functions on [0,1]d with an error O(r-1/d), where boldsymbolg is realized by a fixed-size ReLU network, boldsymbolL1 and L2 are two affine linear maps matching the dimensions, and boldsymbolgcirc r denotes the r-times composition of boldsymbolg. Furthermore, we extend such a result to generic continuous functions on [0,1]d with the approximation error characterized by the modulus of continuity. Our results reveal that a continuous-depth network generated via a dynamical system has immense approximation power even if its dynamics function is time-independent and realized by a fixed-size ReLU network.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 29 Jan 2023.

Builds on

29 of 48 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  4. 2016Error bounds for approximations with deep ReLU networkscited by 567
  5. 2018Universality of Deep Convolutional Neural Networkscited by 176
  6. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  7. 2020Deep Network Approximation for Smooth Functionscited by 163
  8. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  9. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  10. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  11. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
  12. 2019The phase diagram of approximation rates for deep neural networkscited by 86
17 more
  1. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  2. 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
  3. 2019Approximation spaces of deep neural networkscited by 51
  4. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  5. 2022Task Adaptive Parameter Sharing for Multi-Task Learningcited by 42
  6. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  7. 2019Nonlinear Approximation via Compositionscited by 37
  8. 2019Learning Implicitly Recurrent CNNs Through Parameter Sharingcited by 27
  9. 2018Approximation and Estimation for High-Dimensional Deep Learning Networkscited by 22
  10. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  11. 2020Neural Parameter Allocation Searchcited by 15
  12. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  13. 2022Deep Neural Network Approximation of Invariant Functions through Dynamical Systemscited by 8
  14. 2018Construction of neural networks for realization of localized deep learningcited by 7
  15. 2021Rethinking Hard-Parameter Sharing in Multi-Domain Learningcited by 6
  16. 2022On the Universal Approximation Property of Deep Fully Convolutional Neural Networkscited by 6
  17. 2020ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolutioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 1.

6 papers.

  1. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  2. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  3. 2024Structured and Balanced Multi-Component and Multi-Layer Neural Networkscited by 3
  4. 2025Self-composing neural operators for high-frequency and multiscale PDE surrogatescited by 3
  5. 2026Multigrade Neural Network Approximationcited by 1
  6. 2025NeuronSeek: On Stability and Expressivity of Task-driven Neuronscited by 0