Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid
Not on the map. Why?
Abstract (arXiv)
A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jul 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 5; 2022: 2; 2023: 1.
8 papers.
- 2021Differentiable Multiple Shooting Layerscited by 20
- 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
- 2021Globally Convergent Multilevel Training of Deep Residual Networkscited by 11
- 2021GIST: Distributed Training for Large-Scale Graph Convolutional Networkscited by 10
- 2023DeepPCR: Parallelizing Sequential Operations in Neural Networkscited by 9
- 2022Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequencescited by 5
- 2021Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networkscited by 2
- 2022Multilevel-in-Layer Training for Deep Neural Network Regressioncited by 0
- 2023DeepPCR: Parallelizing Sequential Operations in Neural Networks28 Sep 2023
- 2022Multilevel-in-Layer Training for Deep Neural Network Regression11 Nov 2022
- 2022Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences7 Mar 2022
- 2021Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks10 Dec 2021
- 2021Globally Convergent Multilevel Training of Deep Residual Networks15 Jul 2021
- 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Training2 Jul 2021
- 2021Differentiable Multiple Shooting Layers7 Jun 2021
- 2021GIST: Distributed Training for Large-Scale Graph Convolutional Networks20 Feb 2021