# Multilevel Minimization for Deep Residual Networks

arXiv: https://arxiv.org/abs/2004.06196 · HTML: https://mlsanity.com/paper/2004.06196/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies](https://mlsanity.com/paper/2306.17648/) (2023)
- [Globally Convergent Multilevel Training of Deep Residual Networks](https://mlsanity.com/paper/2107.07572/) (2021)
- [Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training](https://mlsanity.com/paper/2302.07049/) (2023)
- [Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications](https://mlsanity.com/paper/2406.10997/) (2024)
- [Spline parameterization of neural network controls for deep learning](https://mlsanity.com/paper/2103.00301/) (2021)
- [Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization](https://mlsanity.com/paper/2507.11513/) (2025)
- [Sparse Training of Neural Networks based on Multilevel Mirror Descent](https://mlsanity.com/paper/2602.03535/) (2026)
- [Training of deep residual networks with stochastic MG/OPT](https://mlsanity.com/paper/2108.04052/) (2021)
- [Multilevel Training for Kolmogorov Arnold Networks](https://mlsanity.com/paper/2603.04827/) (2026)
- [A Multi-Level Framework for Accelerating Training Transformer Models](https://mlsanity.com/paper/2404.07999/) (2024)
- [A multilevel approach to accelerate the training of Transformers](https://mlsanity.com/paper/2504.18590/) (2025)

## Abstract (arXiv)

We present a new multilevel minimization framework for the training of deep residual networks (ResNets), which has the potential to significantly reduce training time and effort. Our framework is based on the dynamical system's viewpoint, which formulates a ResNet as the discretization of an initial value problem. The training process is then formulated as a time-dependent optimal control problem, which we discretize using different time-discretization parameters, eventually generating multilevel-hierarchy of auxiliary networks with different resolutions. The training of the original ResNet is then enhanced by training the auxiliary networks with reduced resolutions. By design, our framework is conveniently independent of the choice of the training strategy chosen on each level of the multilevel hierarchy. By means of numerical examples, we analyze the convergence behavior of the proposed method and demonstrate its robustness. For our examples we employ a multilevel gradient-based methods. Comparisons with standard single level methods show a speedup of more than factor three while achieving the same validation accuracy.