mlsanityData to 14 Oct 2026

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

arXiv 1910.02054 ↗First posted 4 Oct 2019cs.LG cs.DC stat.MLcited by 1,301

Not on the map. Why?

Cited by 109 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large deep learning models offer significant accuracy gains, but training billions to trillions of parameters is challenging. Existing solutions such as data and model parallelisms exhibit fundamental limitations to fit these models into limited device memory, while obtaining computation, communication and development efficiency. We develop a novel solution, Zero Redundancy Optimizer (ZeRO), to optimize memory, vastly improving training speed while increasing the model size that can be efficiently trained. ZeRO eliminates memory redundancies in data- and model-parallel training while retaining low communication volume and high computational granularity, allowing us to scale the model size proportional to the number of devices with sustained high efficiency. Our analysis on memory requirements and communication volume demonstrates: ZeRO has the potential to scale beyond 1 Trillion parameters using today's hardware. We implement and evaluate ZeRO: it trains large models of over 100B parameter with super-linear speedup on 400 GPUs, achieving throughput of 15 Petaflops. This represents an 8x increase in model size and 10x increase in achievable performance over state-of-the-art. In terms of usability, ZeRO can train large models of up to 13B parameters (e.g., larger than Megatron GPT 8.3B and T5 11B) without requiring model parallelism which is harder for scientists to apply. Last but not the least, researchers have used the system breakthroughs of ZeRO to create the world's largest language model (Turing-NLG, 17B parameters) with record breaking accuracy.

Key facts. Cited by 1,301 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 18; 2023: 81; 2024: 350; 2025: 439; 2026: 405.

1,301 papers.

Sort:

By area:

  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  10. 2023A Comprehensive Overview of Large Language Modelscited by 648
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 613
38 more
  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  2. 2025Kimi K2: Open Agentic Intelligencecited by 567
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  4. 2022Efficiently Scaling Transformer Inferencecited by 539
  5. 2024Large Language Models: A Surveycited by 525
  6. 2024Long Context Transfer from Language to Visioncited by 515
  7. 2025LIMO: Less is More for Reasoningcited by 498
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  10. 2025Muon is Scalable for LLM Trainingcited by 428
  11. 2023Llemma: An Open Language Model For Mathematicscited by 388
  12. 2025Kimi-VL Technical Reportcited by 388
  13. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  14. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  15. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  16. 2024LoRA Learns Less and Forgets Lesscited by 328
  17. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  18. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  19. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  20. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  21. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  22. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  23. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  24. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  25. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  26. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  27. 2025Training Language Models to Reason Efficientlycited by 247
  28. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  29. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  30. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  31. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  32. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  33. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  34. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  35. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  36. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  37. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 165
  38. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163

50 most cited of 1,301 All 1,301 (JSON)