mlsanityData to 7 Oct 2026

Globally Convergent Multilevel Training of Deep Residual Networks

arXiv 2107.07572 ↗First posted 15 Jul 2021cs.LG cs.NA math.NAcited by 11

Not on the map. Why?

Abstract (arXiv)

We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trust-region (RMTR) method, which operates in hybrid (stochastic-deterministic) settings by adaptively adjusting mini-batch sizes during the training. The multilevel hierarchy and the transfer operators are constructed by exploiting a dynamical system's viewpoint, which interprets forward propagation through the ResNet as a forward Euler discretization of an initial value problem. In contrast to traditional training approaches, our novel RMTR method also incorporates curvature information on all levels of the multilevel hierarchy by means of the limited-memory SR1 method. The overall performance and the convergence properties of our multilevel training method are numerically investigated using examples from the field of classification and regression.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 38 of its 86 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

38 of 86 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2016Deep Networks with Stochastic Depthcited by 973
  10. 2017Stable Architectures for Deep Neural Networkscited by 340
  11. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  12. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
26 more
  1. 2020Dissecting Neural ODEscited by 150
  2. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  3. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  4. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 68
  5. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  6. 2019A Multigrid Method for Efficiently Training Video Modelscited by 52
  7. 2017Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Informationcited by 41
  8. 2020Continuous-in-Depth Neural Networkscited by 40
  9. 2017Second-Order Optimization for Non-Convex Machine Learning: An Empirical Studycited by 35
  10. 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
  11. 2017Adaptive Sampling Strategies for Stochastic Optimizationcited by 29
  12. 2015Stochastic Optimization Using a Trust-Region Method and Random Modelscited by 25
  13. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
  14. 2016Stochastic Block BFGS: Squeezing More Curvature out of Datacited by 24
  15. 2020Multilevel Minimization for Deep Residual Networkscited by 11
  16. 2018Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximationscited by 8
  17. 2020Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigridcited by 8
  18. 2017A Robust Multi-Batch L-BFGS Method for Machine Learningcited by 7
  19. 2019Multilevel Initialization for Layer-Parallel Deep Neural Network Trainingcited by 7
  20. 2015On Solving L-SR1 Trust-Region Subproblemscited by 6
  21. 2016Block BFGS Methodscited by 3
  22. 2021Spline parameterization of neural network controls for deep learningcited by 3
  23. 2019Recursive multilevel trust region method with application to fully monolithic phase-field models of brittle fracturecited by 2
  24. 2019On the approximation of the solution of partial differential equations by artificial neural networks trained by a multilevel Levenberg-Marquardt methodcited by 1
  25. 2019An Adaptive Sample Size Trust-Region Method for Finite-Sum Minimizationcited by 1
  26. 2020Large scale simulation of pressure induced phase-field fracture propagation using Utopiacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 0; 2025: 2; 2026: 4.

11 papers.

  1. 2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
  2. 2023A Block-Coordinate Approach of Multi-level Optimization with an Application to Physics-Informed Neural Networkscited by 7
  3. 2023Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Trainingcited by 6
  4. 2025Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimizationcited by 3
  5. 2026Sparse Training of Neural Networks based on Multilevel Mirror Descentcited by 3
  6. 2023Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methodscited by 2
  7. 2023A Domain Decomposition-Based CNN-DNN Architecture for Model Parallel Training Applied to Image Recognition Problemscited by 1
  8. 2026Multilevel Training for Kolmogorov Arnold Networkscited by 1
  9. 2025A multilevel approach to accelerate the training of Transformerscited by 0
  10. 2026A Non-Monotone Preconditioned Trust-Region Method for Neural Network Trainingcited by 0
  11. 2026Two-level domain-decomposition AdaGrad method for scalable training of graph neural networkscited by 0