mlsanityData to 14 Oct 2026

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

arXiv 2104.07857 ↗First posted 16 Apr 2021cs.DC cs.AI cs.LGcited by 256

Not on the map. Why?

Cited by 29 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model. In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaflops on 512 NVIDIA V100 GPUs(40% of peak), while also demonstrating super linear scalability. An open source implementation of ZeRO-Infinity is available through DeepSpeed, a deep learning optimization library that makes distributed training easy, efficient, and effective.

Key facts. Cited by 256 papers in the map (data through 14 Oct 2026). Builds on 20 of its 44 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

20 of 44 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
8 more
  1. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  2. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  3. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  4. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  5. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  6. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
  7. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  8. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 31; 2023: 36; 2024: 73; 2025: 54; 2026: 38.

256 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  7. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  10. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  12. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
38 more
  1. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  2. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  3. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  4. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  5. 2023Efficient Large Language Models: A Surveycited by 186
  6. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  7. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  9. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  10. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  11. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  12. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  13. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  14. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  15. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  16. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  17. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  18. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  19. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 93
  20. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  21. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  22. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  23. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  24. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  25. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  26. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 70
  27. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  28. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  29. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  30. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  31. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 57
  32. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  33. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  34. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  35. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  36. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  37. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  38. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47

50 most cited of 256 All 256 (JSON)