# MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

arXiv: https://arxiv.org/abs/2205.00119 · HTML: https://mlsanity.com/paper/2205.00119/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2022.

## Built on by (most cited first, 22 of 22)

- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [ZeRO++: Extremely Efficient Collective Communication for Giant Model Training](https://mlsanity.com/paper/2306.10209/) (2023)
- [Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity](https://mlsanity.com/paper/2309.10285/) (2023)
- [An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training](https://mlsanity.com/paper/2312.11819/) (2023)
- [DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines](https://mlsanity.com/paper/2311.10418/) (2023)
- [Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training](https://mlsanity.com/paper/2302.08005/) (2023)
- [Zero redundancy distributed learning with differential privacy](https://mlsanity.com/paper/2311.11822/) (2023)
- [Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation](https://mlsanity.com/paper/2403.00877/) (2024)
- [MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training](https://mlsanity.com/paper/2407.12117/) (2024)
- [Rethinking Memory and Communication Cost for Efficient Large Language Model Training](https://mlsanity.com/paper/2310.06003/) (2023)
- [Large Scale Generative AI Text Applied to Sports and Music](https://mlsanity.com/paper/2402.15514/) (2024)
- [Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models](https://mlsanity.com/paper/2411.10003/) (2024)
- [DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster](https://mlsanity.com/paper/2506.21263/) (2025)
- [Adaptive Two-Stage Cloud Resource Scaling via Hierarchical Multi-Indicator Forecasting and Bayesian Decision-Making](https://mlsanity.com/paper/2408.01000/) (2024)
- [RAF: Holistic Compilation for Deep Learning Model Training](https://mlsanity.com/paper/2303.04759/) (2023)
- [Just-in-time Quantization with Processing-In-Memory for Efficient ML Training](https://mlsanity.com/paper/2311.05034/) (2023)
- [Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks](https://mlsanity.com/paper/2403.08837/) (2024)
- [Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning](https://mlsanity.com/paper/2501.04266/) (2025)
- [CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training](https://mlsanity.com/paper/2604.24013/) (2026)
- [BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training](https://mlsanity.com/paper/2609.03151/) (2026)

## Abstract (arXiv)

Existing general purpose frameworks for gigantic model training, i.e., dense models with billions of parameters, cannot scale efficiently on cloud environment with various networking conditions due to large communication overheads. In this paper, we propose MiCS, which Minimizes the Communication Scale to bring down communication overhead. Specifically, by decreasing the number of participants in a communication collective, MiCS can utilize heterogeneous network bandwidth, reduce network traffic over slower links, reduce the latency of communications for maintaining high network bandwidth utilization, and amortize expensive global gradient synchronization overhead. Our evaluation on AWS shows that the system throughput of MiCS is up to 2.89× that of the state-of-the-art large model training systems. MiCS achieves near-linear scaling efficiency, which is up to 1.27× that of DeepSpeed. MiCS allows us to train a proprietary model with 100 billion parameters on 512 GPUs with 99.4% weak-scaling efficiency, and it is able to saturate over 54.5% theoretical computation power of each GPU on a public cloud with less GPU memory and more restricted networks than DGX-A100 clusters.