mlsanityData to 14 Oct 2026

ZeRO++: Extremely Efficient Collective Communication for Giant Model Training

arXiv 2306.10209 ↗First posted 16 Jun 2023cs.DC cs.AI cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

Zero Redundancy Optimizer (ZeRO) has been used to train a wide range of large language models on massive GPUs clusters due to its ease of use, efficiency, and good scalability. However, when training on low-bandwidth clusters, or at scale which forces batch size per GPU to be small, ZeRO's effective throughput is limited because of high communication volume from gathering weights in forward pass, backward pass, and averaging gradients. This paper introduces three communication volume reduction techniques, which we collectively refer to as ZeRO++, targeting each of the communication collectives in ZeRO. First is block-quantization based all-gather. Second is data remapping that trades-off communication for more memory. Third is a novel all-to-all based quantized gradient averaging paradigm as replacement of reduce-scatter collective, which preserves accuracy despite communicating low precision data. Collectively, ZeRO++ reduces communication volume of ZeRO by 4x, enabling up to 2.16x better throughput at 384 GPU scale.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 18 of its 39 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

18 of 39 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  8. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  9. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  10. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  11. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  12. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
6 more
  1. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  2. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  3. 2022Datasheet for the Pilecited by 43
  4. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  5. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  6. 2022MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloudcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 21; 2025: 13; 2026: 11.

52 papers.

Sort:

By area:

  1. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  2. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  3. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 78
  4. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  5. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  6. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  7. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  8. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  9. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  10. 2025WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Trainingcited by 17
  11. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  12. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 14
38 more
  1. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  2. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  3. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  4. 2024WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Averagecited by 9
  5. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  6. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  7. 2024Training Language Models on Synthetic Edit Sequences Improves Code Synthesiscited by 7
  8. 2023diff History for Neural Language Agentscited by 6
  9. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  10. 2024LoCo: Low-Bit Communication Adaptor for Large-scale Model Trainingcited by 6
  11. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
  12. 2025FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarioscited by 5
  13. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 5
  14. 2023Rethinking Memory and Communication Cost for Efficient Large Language Model Trainingcited by 4
  15. 2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
  16. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  17. 2023LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Modelscited by 2
  18. 2024Accelerating Large Language Model Training with Hybrid GPU-based Compressioncited by 2
  19. 2024PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignmentcited by 2
  20. 2025Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systemscited by 2
  21. 2025EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Accelerationcited by 2
  22. 2025Towards Decentralized and Sustainable Foundation Model Training with the Edgecited by 2
  23. 2025Improving LLMs for Machine Translation Using Synthetic Preference Datacited by 2
  24. 2026UCCL-Zip: Lossless Compression Supercharged GPU Communicationcited by 2
  25. 2023Just-in-time Quantization with Processing-In-Memory for Efficient ML Trainingcited by 1
  26. 2024Enhancing Stability for Large Language Models Training in Constrained Bandwidth Networkscited by 1
  27. 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communicationcited by 1
  28. 2026FastUSP: A Multi-Level Collaborative Acceleration Framework for Distributed Diffusion Model Inferencecited by 1
  29. 2026LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Trainingcited by 1
  30. 2024LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reductioncited by 0
  31. 2024Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networkscited by 0
  32. 2024FedProphet: Memory-Efficient Federated Adversarial Training via Robust and Consistent Cascade Learningcited by 0
  33. 2024Photon: Federated LLM Pre-Trainingcited by 0
  34. 2025Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioningcited by 0
  35. 2026Horizon-LM: A RAM-Centric Architecture for LLM Trainingcited by 0
  36. 2026A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Networkcited by 0
  37. 2026FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Costcited by 0
  38. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026NCCLZ: Compression-Enabled GPU Collectives with Decoupled Quantization and Entropy Coding
  2. 2026ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
  3. 2025AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training
  4. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  5. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  6. 2026Fast NF4 Dequantization Kernels for Large Language Model Inference
  7. 2025ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
  8. 2024Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference