mlsanityData to 14 Oct 2026

Dynamic Tensor Rematerialization

arXiv 2006.09616 ↗First posted 17 Jun 2020cs.LG cs.PL stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

Checkpointing enables the training of deep learning models under restricted memory budgets by freeing intermediate activations from memory and recomputing them on demand. Current checkpointing techniques statically plan these recomputations offline and assume static computation graphs. We demonstrate that a simple online algorithm can achieve comparable performance by introducing Dynamic Tensor Rematerialization (DTR), a greedy online algorithm for checkpointing that is extensible and general, is parameterized by eviction policy, and supports dynamic models. We prove that DTR can train an N-layer linear feedforward network on an Omega(√N) memory budget with only O(N) tensor operations. DTR closely matches the performance of optimal static checkpointing in simulated experiments. We incorporate a DTR prototype into PyTorch merely by interposing on tensor allocations and operator calls and collecting lightweight metadata on tensors.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 16; 2024: 2; 2025: 5; 2026: 8.

46 papers.

Sort:

By area:

  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  2. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  3. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  4. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  5. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  6. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 39
  7. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  8. 2022Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programscited by 28
  9. 2022POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Pagingcited by 26
  10. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  11. 2022On-device Training: A First Overview on Existing Systemscited by 19
  12. 2023TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edgecited by 18
34 more
  1. 2021The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Paddingcited by 17
  2. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  3. 2023Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Trainingcited by 11
  4. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resourcescited by 10
  5. 2023Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Modelcited by 9
  6. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  7. 2023Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devicescited by 7
  8. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  9. 2025SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Trainingcited by 6
  10. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
  11. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  12. 2023SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamicscited by 5
  13. 2024MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Trainingcited by 5
  14. 2022Survey on Large Scale Neural Network Trainingcited by 4
  15. 2022Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reductioncited by 4
  16. 2023Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorchcited by 4
  17. 2024SoD2: Statically Optimizing Dynamic Deep Neural Networkcited by 4
  18. 2023Coop: Memory is not a Commoditycited by 3
  19. 2022DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputationcited by 2
  20. 2022XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environmentscited by 2
  21. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  22. 2023RAF: Holistic Compilation for Deep Learning Model Trainingcited by 1
  23. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  24. 2026AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelismcited by 1
  25. 2026Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUscited by 1
  26. 2026Deterministic Regime Switching and Feasibility Inversion in Dynamic Tensor Rematerializationcited by 1
  27. 2023An Evaluation of Memory Optimization Methods for Training Neural Networkscited by 0
  28. 2023Moccasin: Efficient Tensor Rematerialization for Neural Networkscited by 0
  29. 2025GPU Memory Usage Optimization for Backward Propagation in Deep Network Trainingcited by 0
  30. 2026TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditchingcited by 0
  31. 2026TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modelingcited by 0
  32. 2026Flattening Every Memory Peak in Long-Context Mixture-of-Experts Trainingcited by 0
  33. 2026FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attentioncited by 0
  34. 2026When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerializationcited by 0