# LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentation

arXiv: https://arxiv.org/abs/2006.12575 · HTML: https://mlsanity.com/paper/2006.12575/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Test-Time Training for Deformable Multi-Scale Image Registration](https://mlsanity.com/paper/2103.13578/) (2021)
- [GIST: Distributed Training for Large-Scale Graph Convolutional Networks](https://mlsanity.com/paper/2102.10424/) (2021)
- [Systems for Parallel and Distributed Large-Model Deep Learning Training](https://mlsanity.com/paper/2301.02691/) (2023)
- [HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2412.11458/) (2024)
- [LOFT: Finding Lottery Tickets through Filter-wise Training](https://mlsanity.com/paper/2210.16169/) (2022)
- [Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation](https://mlsanity.com/paper/2110.15884/) (2021)
- [Open-Source Skull Reconstruction with MONAI](https://mlsanity.com/paper/2211.14051/) (2022)

## Abstract (arXiv)

Deep Learning (DL) models are becoming larger, because the increase in model size might offer significant accuracy gain. To enable the training of large deep networks, data parallelism and model parallelism are two well-known approaches for parallel training. However, data parallelism does not help reduce memory footprint per device. In this work, we introduce Large deep 3D ConvNets with Automated Model Parallelism (LAMP) and investigate the impact of both input's and deep 3D ConvNets' size on segmentation accuracy. Through automated model parallelism, it is feasible to train large deep 3D ConvNets with a large input patch, even the whole image. Extensive experiments demonstrate that, facilitated by the automated model parallelism, the segmentation accuracy can be improved through increasing model size and input context size, and large input yields significant inference speedup compared with sliding window of small patches in the inference. Code is availablefootnotehttps://monai.io/research/lamp-automated-model-parallelism.