mlsanityData to 7 Oct 2026

Convergence of Deep ReLU Networks

arXiv 2107.12530 ↗First posted 27 Jul 2021cs.LG math.FAcited by 14

Not on the map. Why?

Abstract (arXiv)

We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to the design strategy of the well-known deep residual networks in image classification.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

23 of 39 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2016Error bounds for approximations with deep ReLU networkscited by 567
  5. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  6. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  8. 2016Deep Information Propagationcited by 194
  9. 2018Universality of Deep Convolutional Neural Networkscited by 176
  10. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  11. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  12. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
11 more
  1. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  2. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  3. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  4. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  5. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  6. 2019Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operatorscited by 54
  7. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  8. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  9. 2018On Lipschitz Bounds of General Convolutional Neural Networkscited by 29
  10. 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
  11. 2020A Mathematical Introduction to Generative Adversarial Nets (GAN)cited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 3; 2025: 0; 2026: 3.

14 papers.

Sort:

  1. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14
  2. 2021Convergence of Deep Convolutional Neural Networkscited by 13
  3. 2023Multi-Grade Deep Learningcited by 12
  4. 2022Sparse Deep Neural Network for Nonlinear Partial Differential Equationscited by 5
  5. 2023Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widthscited by 5
  6. 2024Deep Neural Network Solutions for Oscillatory Fredholm Integral Equationscited by 5
  7. 2023Successive Affine Learning for Deep Neural Networkscited by 4
  8. 2024Hypothesis Spaces for Deep Learningcited by 4
  9. 2026Regime Change Hypothesis: Foundations for Decoupled Dynamics in Neural Network Trainingcited by 3
  10. 2023Classification of integers based on residue classes via modern deep learning algorithmscited by 2
  11. 2022Convergence Analysis of Deep Residual Networkscited by 1
  12. 2022Convergence of Deep Neural Networks with General Activation Functions and Poolingcited by 0
2 more
  1. 2026Detecting Atypical Clients in Federated Learning via Representation-Level Divergencecited by 0
  2. 2026Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulascited by 0