mlsanityData to 7 Oct 2026

Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint

arXiv 2206.04569 ↗First posted 9 Jun 2022stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Overparameterized neural networks enjoy great representation power on complex data, and more importantly yield sufficiently smooth output, which is crucial to their generalization and robustness. Most existing function approximation theories suggest that with sufficiently many parameters, neural networks can well approximate certain classes of functions in terms of the function value. The neural network themselves, however, can be highly nonsmooth. To bridge this gap, we take convolutional residual networks (ConvResNets) as an example, and prove that large ConvResNets can not only approximate a target function in terms of function value, but also exhibit sufficient first-order smoothness. Moreover, we extend our theory to approximating functions supported on a low-dimensional manifold. Our theory partially justifies the benefits of using deep and wide networks in practice. Numerical experiments on adversarial robust image classification are provided to support our theory.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 70 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

34 of 70 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2016Wide Residual Networkscited by 3,057
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  9. 2016Error bounds for approximations with deep ReLU networkscited by 567
  10. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  11. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  12. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
22 more
  1. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  2. 2018Universality of Deep Convolutional Neural Networkscited by 176
  3. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  4. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  5. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  6. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  7. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  8. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  9. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  10. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  11. 2015Provable approximation properties for deep neural networkscited by 60
  12. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  13. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  14. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  15. 2019Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networkscited by 40
  16. 2018Equivalence of approximation by convolutional neural networks and fully-connected networkscited by 38
  17. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  18. 2016Deep nets for local manifold learningcited by 26
  19. 2020A deep network construction that adapts to intrinsic dimensionality beyond the domaincited by 10
  20. 2019Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionalitycited by 9
  21. 2020Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networkscited by 7
  22. 2021Adversarially Regularized Policy Learning Guided by Trajectory Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 0; 2026: 1.

6 papers.

  1. 2023Deep Nonparametric Estimation of Intrinsic Data Structures by Chart Autoencoders: Generalization Error and Robustnesscited by 10
  2. 2023Double-well Net for Image Segmentationcited by 7
  3. 2023Connections between Operator-splitting Methods and Deep Neural Networks with Applications in Image Segmentationcited by 3
  4. 2024Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimationcited by 3
  5. 2024Can overfitted deep neural networks in adversarial training generalize? -- An approximation viewpointcited by 1
  6. 2026Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activationscited by 0