# Training of deep residual networks with stochastic MG/OPT

arXiv: https://arxiv.org/abs/2108.04052 · HTML: https://mlsanity.com/paper/2108.04052/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [Multilevel Minimization for Deep Residual Networks](https://mlsanity.com/paper/2004.06196/) (2020)
- [DSD: Dense-Sparse-Dense Training for Deep Neural Networks](https://mlsanity.com/paper/1607.04381/) (2016)
- [Multilevel Initialization for Layer-Parallel Deep Neural Network Training](https://mlsanity.com/paper/1912.08974/) (2019)
- [The Power of Sparsity in Convolutional Neural Networks](https://mlsanity.com/paper/1702.06257/) (2017)

## Built on by (most cited first, 2 of 2)

- [A Block-Coordinate Approach of Multi-level Optimization with an Application to Physics-Informed Neural Networks](https://mlsanity.com/paper/2305.14477/) (2023)
- [Multilevel-in-Layer Training for Deep Neural Network Regression](https://mlsanity.com/paper/2211.06515/) (2022)

## Abstract (arXiv)

We train deep residual networks with a stochastic variant of the nonlinear multigrid method MG/OPT. To build the multilevel hierarchy, we use the dynamical systems viewpoint specific to residual networks. We report significant speed-ups and additional robustness for training MNIST on deep residual networks. Our numerical experiments also indicate that multilevel training can be used as a pruning technique, as many of the auxiliary networks have accuracies comparable to the original network.