mlsanityData to 14 Oct 2026

Learning Deep ResNet Blocks Sequentially using Boosting Theory

arXiv 1706.04964 ↗First posted 15 Jun 2017cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep residual network (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct T weak module classifiers, each contains two of the T layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, BoostResNet, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of T "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with l1 norm bounded weights.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 8; 2024: 4; 2025: 2; 2026: 3.

32 papers.

Sort:

By area:

  1. 2021Ensemble deep learning: A reviewcited by 284
  2. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  3. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 60
  4. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  5. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  6. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  7. 2022Building Robust Ensembles via Margin Boostingcited by 8
  8. 2022Boosted Dynamic Neural Networkscited by 8
  9. 2024End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Trainingcited by 8
  10. 2021Boosting Offline Reinforcement Learning with Residual Generative Modelingcited by 7
  11. 2021Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learningcited by 6
  12. 2021Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networkscited by 6
20 more
  1. 2021An Interpretive Constrained Linear Model for ResNet and MgNetcited by 6
  2. 2021Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methodscited by 5
  3. 2022Local Learning on Transformers via Feature Reconstructioncited by 4
  4. 2023Free energy of Bayesian Convolutional Neural Network with Skip Connectioncited by 4
  5. 2023Module-wise Training of Neural Networks via the Minimizing Movement Schemecited by 4
  6. 2025Random Feature Representation Boostingcited by 4
  7. 2023A Gradient Boosting Approach for Training Convolutional and Deep Neural Networkscited by 2
  8. 2022BackLink: Supervised Local Training with Backward Linkscited by 1
  9. 2022Block-wise Training of Residual Networks via the Minimizing Movement Schemecited by 1
  10. 2023AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codescited by 1
  11. 2024ColA: Collaborative Adaptation with Gradient Learningcited by 1
  12. 2025On residual network depthcited by 1
  13. 2023HCE: Improving Performance and Efficiency with Heterogeneously Compressed Neural Network Ensemblecited by 0
  14. 2023Local Boosting for Weakly-Supervised Learningcited by 0
  15. 2023Iterated Piecewise Affine (IPA) Approximation for Language Modelingcited by 0
  16. 2023Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supplycited by 0
  17. 2024Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Servicescited by 0
  18. 2026Gradient Boosting within a Single Attention Layercited by 0
  19. 2026Can an MLP Absorb Its Own Skip Connection Exactly?cited by 0
  20. 2026BoostLoRA: Growing Effective Rank by Boosting Adapterscited by 0