# Learning Deep ResNet Blocks Sequentially using Boosting Theory

arXiv: https://arxiv.org/abs/1706.04964 · HTML: https://mlsanity.com/paper/1706.04964/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2017.

## Built on by (most cited first, 30 of 32)

- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks](https://mlsanity.com/paper/2109.02832/) (2021)
- [Building Robust Ensembles via Margin Boosting](https://mlsanity.com/paper/2206.03362/) (2022)
- [Boosted Dynamic Neural Networks](https://mlsanity.com/paper/2211.16726/) (2022)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Boosting Offline Reinforcement Learning with Residual Generative Modeling](https://mlsanity.com/paper/2106.10411/) (2021)
- [Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning](https://mlsanity.com/paper/2106.06401/) (2021)
- [Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks](https://mlsanity.com/paper/2109.15089/) (2021)
- [An Interpretive Constrained Linear Model for ResNet and MgNet](https://mlsanity.com/paper/2112.07441/) (2021)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Local Learning on Transformers via Feature Reconstruction](https://mlsanity.com/paper/2212.14215/) (2022)
- [Free energy of Bayesian Convolutional Neural Network with Skip Connection](https://mlsanity.com/paper/2307.01417/) (2023)
- [Module-wise Training of Neural Networks via the Minimizing Movement Scheme](https://mlsanity.com/paper/2309.17357/) (2023)
- [Random Feature Representation Boosting](https://mlsanity.com/paper/2501.18283/) (2025)
- [A Gradient Boosting Approach for Training Convolutional and Deep Neural Networks](https://mlsanity.com/paper/2302.11327/) (2023)
- [BackLink: Supervised Local Training with Backward Links](https://mlsanity.com/paper/2205.07141/) (2022)
- [Block-wise Training of Residual Networks via the Minimizing Movement Scheme](https://mlsanity.com/paper/2210.00949/) (2022)
- [AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes](https://mlsanity.com/paper/2308.07221/) (2023)
- [ColA: Collaborative Adaptation with Gradient Learning](https://mlsanity.com/paper/2404.13844/) (2024)
- [On residual network depth](https://mlsanity.com/paper/2510.03470/) (2025)
- [HCE: Improving Performance and Efficiency with Heterogeneously Compressed Neural Network Ensemble](https://mlsanity.com/paper/2301.07794/) (2023)
- [Local Boosting for Weakly-Supervised Learning](https://mlsanity.com/paper/2306.02859/) (2023)
- [Iterated Piecewise Affine (IPA) Approximation for Language Modeling](https://mlsanity.com/paper/2306.12317/) (2023)
- [Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply](https://mlsanity.com/paper/2312.07636/) (2023)
- [Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services](https://mlsanity.com/paper/2408.12526/) (2024)
- [Gradient Boosting within a Single Attention Layer](https://mlsanity.com/paper/2604.03190/) (2026)

## Abstract (arXiv)

Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep residual network (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct T weak module classifiers, each contains two of the T layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, BoostResNet, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of T "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with l1 norm bounded weights.