Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks
Not on the map. Why?
Abstract (arXiv)
Convolutional residual neural networks (ConvResNets), though overparameterized, can achieve remarkable prediction performance in practice, which cannot be well explained by conventional wisdom. To bridge this gap, we study the performance of ConvResNeXts, which cover ConvResNets as a special case, trained with weight decay from the perspective of nonparametric classification. Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks. Specifically, we consider a smooth target function supported on a low-dimensional manifold, then prove that ConvResNeXts can adapt to the function smoothness and low-dimensional structures and efficiently learn the function without suffering from the curse of dimensionality. Our findings partially justify the advantage of overparameterized ConvResNeXts over conventional machine learning models.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 49 references that are in the map. First posted on arXiv 4 Jul 2023.
Builds on
25 of 49 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2017Squeeze-and-Excitation Networkscited by 4,763
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
13 more
- 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
- 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
- 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
- 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
- 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
- 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
- 2019Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networkscited by 40
- 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
- 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
- 2020Sharp Rate of Convergence for Deep Neural Network Classifiers under the Teacher-Student Settingcited by 11
- 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
- 2022Deep Neural Network Classifier for Multi-dimensional Functional Datacited by 5
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.