mlsanityData to 7 Oct 2026

Training Deep Nets with Sublinear Memory Cost

arXiv 1604.06174 ↗First posted 21 Apr 2016cs.LGcited by 825

Not on the map. Why?

Cited by 64 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a systematic approach to reduce the memory consumption of deep neural network training. Specifically, we design an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch. As many of the state-of-the-art models hit the upper bound of the GPU memory, our algorithm allows deeper and more complex models to be explored, and helps advance the innovations in deep learning research. We focus on reducing the memory cost to store the intermediate feature maps and gradients during training. Computation graph analysis is used for automatic in-place operation and memory sharing optimizations. We show that it is possible to trade computation for memory - giving a more memory efficient training algorithm with a little extra computation cost. In the extreme case, our analysis also shows that the memory consumption can be reduced to O(log n) with as little as O(n log n) extra cost for forward computation. Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.

Key facts. Cited by 825 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 84; 2022: 85; 2023: 150; 2024: 202; 2025: 163; 2026: 140.

825 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  11. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  12. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
38 more
  1. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  8. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  11. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  13. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  14. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  15. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  16. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  17. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  18. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  19. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  20. 2025Kimi-VL Technical Reportcited by 381
  21. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  22. 2023Adapting Language Models to Compress Contextscited by 350
  23. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  24. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  25. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  26. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  27. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  28. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  29. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  30. 2021A Field Guide to Federated Optimizationcited by 312
  31. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  32. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 299
  33. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  34. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
  35. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  36. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 237
  37. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  38. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228

50 most cited of 825 All 825 (JSON)