mlsanityData to 7 Oct 2026

Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization

arXiv 1910.02653 ↗First posted 7 Oct 2019cs.LG cs.CV cs.DCcited by 90

Not on the map. Why?

Abstract (arXiv)

We formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization optimization problem, a generalization of prior checkpointing strategies. We introduce Checkmate, a system that solves for optimal rematerialization schedules in reasonable times (under an hour) using off-the-shelf MILP solvers or near-optimal schedules with an approximation algorithm, then uses these schedules to accelerate millions of training iterations. Our method scales to complex, realistic architectures and is hardware-aware through the use of accelerator-specific, profile-based cost models. In addition to reducing training cost, Checkmate enables real-world networks to be trained with up to 5.1x larger input sizes. Checkmate is an open-source project, available at https://github.com/parasj/checkmate.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 23; 2023: 17; 2024: 12; 2025: 12; 2026: 11.

90 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  4. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  5. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  6. 2024AI and Memory Wallcited by 89
  7. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  8. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  9. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  10. 2023Stateful Large Language Model Serving with Pensievecited by 59
  11. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  12. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
38 more
  1. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  2. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  3. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  4. 2021Accelerating Quadratic Optimization with Reinforcement Learningcited by 30
  5. 2022Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programscited by 28
  6. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 28
  7. 2022POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Pagingcited by 26
  8. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  9. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  10. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  11. 2022On-device Training: A First Overview on Existing Systemscited by 18
  12. 2023TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edgecited by 18
  13. 2021The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Paddingcited by 17
  14. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  15. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  16. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  17. 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13
  18. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Modelscited by 12
  19. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  20. 2023Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Trainingcited by 11
  21. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  22. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resourcescited by 10
  23. 2023Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Modelcited by 9
  24. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  25. 2024Gradient descent with generalized Newton's methodcited by 9
  26. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  27. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  28. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  29. 2022OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networkscited by 6
  30. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 6
  31. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  32. 2025SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Trainingcited by 5
  33. 2022Survey on Large Scale Neural Network Trainingcited by 4
  34. 2022Scientometric Review of Artificial Intelligence for Operations & Maintenance of Wind Turbines: The Past, Present and Futurecited by 4
  35. 2022Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reductioncited by 4
  36. 2023SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamicscited by 4
  37. 2023Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorchcited by 4
  38. 2024SoD2: Statically Optimizing Dynamic Deep Neural Networkcited by 4

50 most cited of 90 All 90 (JSON)