# Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

arXiv: https://arxiv.org/abs/2007.07336 · HTML: https://mlsanity.com/paper/2007.07336/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 8 of 8)

- [Differentiable Multiple Shooting Layers](https://mlsanity.com/paper/2106.03885/) (2021)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Globally Convergent Multilevel Training of Deep Residual Networks](https://mlsanity.com/paper/2107.07572/) (2021)
- [GIST: Distributed Training for Large-Scale Graph Convolutional Networks](https://mlsanity.com/paper/2102.10424/) (2021)
- [DeepPCR: Parallelizing Sequential Operations in Neural Networks](https://mlsanity.com/paper/2309.16318/) (2023)
- [Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences](https://mlsanity.com/paper/2203.04738/) (2022)
- [Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks](https://mlsanity.com/paper/2112.05387/) (2021)
- [Multilevel-in-Layer Training for Deep Neural Network Regression](https://mlsanity.com/paper/2211.06515/) (2022)

## Abstract (arXiv)

A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.