mlsanityData to 14 Oct 2026

Generalization Ability of Wide Residual Networks

arXiv 2305.18506 ↗First posted 29 May 2023stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the generalization ability of the wide residual network on mathbbSd-1 with the ReLU activation function. We first show that as the width mrightarrow∞, the residual network kernel (RNK) uniformly converges to the residual neural tangent kernel (RNTK). This uniform convergence further guarantees that the generalization error of the residual network converges to that of the kernel regression with respect to the RNTK. As direct corollaries, we then show i) the wide residual network with the early stopping strategy can achieve the minimax rate provided that the target regression function falls in the reproducing kernel Hilbert space (RKHS) associated with the RNTK; ii) the wide residual network can not generalize well if it is trained till overfitting the data. We finally illustrate some experiments to reconcile the contradiction between our theoretical result and the widely observed ``benign overfitting phenomenon''

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

27 of 40 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2018On Lazy Training in Differentiable Programmingcited by 675
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  10. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  11. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  12. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
15 more
  1. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 89
  2. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  3. 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
  4. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  5. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  6. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  7. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  8. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  9. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 31
  10. 2023Generalization Ability of Wide Neural Networks on ℝcited by 30
  11. 2023On the Optimality of Misspecified Spectral Algorithmscited by 28
  12. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 21
  13. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  14. 2019Towards Understanding the Importance of Shortcut Connections in Residual Networkscited by 18
  15. 2023Kernel interpolation generalizes poorlycited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 9
  2. 2024On the Impacts of the Random Initialization in the Neural Tangent Kernel Theorycited by 5
  3. 2026Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learningcited by 4
  4. 2025Divergence of Empirical Neural Tangent Kernel in Classification Problemscited by 3
  5. 2026DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillationcited by 0