# The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism

arXiv: https://arxiv.org/abs/2007.12856 · HTML: https://mlsanity.com/paper/2007.12856/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2020.

## Built on by (most cited first, 11 of 11)

- [Clairvoyant Prefetching for Distributed Machine Learning I/O](https://mlsanity.com/paper/2101.08734/) (2021)
- [A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks](https://mlsanity.com/paper/2111.04949/) (2021)
- [Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation](https://mlsanity.com/paper/2110.15884/) (2021)
- [Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplars](https://mlsanity.com/paper/2406.17812/) (2024)
- [Improving the Robustness of Federated Learning for Severely Imbalanced Datasets](https://mlsanity.com/paper/2204.13414/) (2022)
- [RAMP: A Flat Nanosecond Optical Network and MPI Operations for Distributed Deep Learning Systems](https://mlsanity.com/paper/2211.15226/) (2022)
- [SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates](https://mlsanity.com/paper/2211.00224/) (2022)
- [A Survey From Distributed Machine Learning to Distributed Deep Learning](https://mlsanity.com/paper/2307.05232/) (2023)
- [Real-time and Downtime-tolerant Fault Diagnosis for Railway Turnout Machines (RTMs) Empowered with Cloud-Edge Pipeline Parallelism](https://mlsanity.com/paper/2411.02086/) (2024)
- [First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training](https://mlsanity.com/paper/2510.14614/) (2025)
- [TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modeling](https://mlsanity.com/paper/2608.15211/) (2026)

## Abstract (arXiv)

We present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.