mlsanityData to 14 Oct 2026

Reducing Activation Recomputation in Large Transformer Models

arXiv 2205.05198 ↗First posted 10 May 2022cs.LG cs.CLcited by 302

Not on the map. Why?

Cited by 32 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training large transformer models is one of the most important computational challenges of modern AI. In this paper, we show how to significantly accelerate training of large transformer models by reducing activation recomputation. Activation recomputation is commonly used to work around memory capacity constraints. Rather than storing activations for backpropagation, they are traditionally recomputed, which saves memory but adds redundant compute. In this work, we show most of this redundant compute is unnecessary because we can reduce memory consumption sufficiently without it. We present two novel yet very simple techniques: sequence parallelism and selective activation recomputation. In conjunction with tensor parallelism, these techniques almost eliminate the need to recompute activations. We evaluate our approach on language models up to one trillion parameters in scale and show that our method reduces activation memory by 5x, while reducing execution time overhead from activation recomputation by over 90%. For example, when training a 530B parameter GPT-3 style model on 2240 NVIDIA A100 GPUs, we achieve a Model Flops Utilization of 54.2%, which is 29% faster than the 42.1% we achieve using recomputation. Our implementation will be available in both Megatron-LM and NeMo-Megatron.

Key facts. Cited by 302 papers in the map (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

17 of 23 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  9. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  10. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  11. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  12. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
5 more
  1. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  2. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  3. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  4. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  5. 2021Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Trainingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 53; 2024: 85; 2025: 94; 2026: 65.

302 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  7. 2025Qwen-Image Technical Reportcited by 1,044
  8. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  9. 2023BloombergGPT: A Large Language Model for Financecited by 939
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  3. 2022Efficiently Scaling Transformer Inferencecited by 539
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  5. 2025Kimi-VL Technical Reportcited by 388
  6. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  7. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  9. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  10. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  12. 2023Efficient Large Language Models: A Surveycited by 186
  13. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  14. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  15. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  16. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  17. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  18. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  19. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  20. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  21. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
  22. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  23. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  24. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  25. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  26. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  27. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
  28. 2024AI and Memory Wallcited by 92
  29. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  30. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  31. 2024Hyper-Connectionscited by 84
  32. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  33. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  34. 2023Stateful Large Language Model Serving with Pensievecited by 61
  35. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61
  36. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  37. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 58
  38. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57

50 most cited of 302 All 302 (JSON)