mlsanityData to 7 Oct 2026

ZeRO-Offload: Democratizing Billion-Scale Model Training

arXiv 2101.06840 ↗First posted 18 Jan 2021cs.DC cs.LGcited by 339

Not on the map. Why?

Cited by 33 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.

Key facts. Cited by 339 papers in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

17 of 39 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  11. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  12. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
5 more
  1. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  2. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  3. 2020Pretrained Language Models for Dialogue Generation with Multiple Input Sourcescited by 14
  4. 2019Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUscited by 10
  5. 2019Efficient Memory Management for GPU-based Deep Learning Systemscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 28; 2023: 69; 2024: 95; 2025: 75; 2026: 46.

339 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
38 more
  1. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  2. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  3. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  4. 2026Gemma 4 Technical Reportcited by 187
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  8. 2023Generative Judge for Evaluating Alignmentcited by 163
  9. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  10. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  11. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  12. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  13. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  14. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  15. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  16. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  17. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  18. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  19. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  20. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  21. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  22. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  23. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  24. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  25. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  26. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  27. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  28. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  29. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  30. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  31. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  32. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  33. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  34. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  35. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  36. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  37. 2025Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent Diffusion Modelscited by 61
  38. 2023Stateful Large Language Model Serving with Pensievecited by 59

50 most cited of 339 All 339 (JSON)