mlsanityData to 7 Oct 2026

Memory-Efficient Backpropagation Through Time

arXiv 1606.03401 ↗First posted 10 Jun 2016cs.NE cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

We propose a novel approach to reduce memory consumption of the backpropagation through time (BPTT) algorithm when training recurrent neural networks (RNNs). Our approach uses dynamic programming to balance a trade-off between caching of intermediate results and recomputation. The algorithm is capable of tightly fitting within almost any user-set memory budget while finding an optimal execution policy minimizing the computational cost. Computational devices have limited memory capacity and maximizing a computational performance given a fixed memory budget is a practical use-case. We provide asymptotic computational upper bounds for various regimes. The algorithm is particularly effective for long sequences. For sequences of length 1000, our algorithm saves 95% of memory usage while using only one third more time per iteration than the standard BPTT.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 0; 2020: 0; 2021: 11; 2022: 14; 2023: 18; 2024: 18; 2025: 10; 2026: 6.

78 papers.

Sort:

By area:

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  3. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  4. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  5. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  6. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  7. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  8. 2022Tutorial on amortized optimizationcited by 83
  9. 2021A Survey on Green Deep Learningcited by 52
  10. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  11. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  12. 2024Tiny Machine Learning: Progress and Futurescited by 40
38 more
  1. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 38
  2. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  3. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  4. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
  5. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  6. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  7. 2024ORBIT: Oak Ridge Base Foundation Model for Earth System Predictabilitycited by 17
  8. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learningcited by 16
  9. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  10. 2023DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Trainingcited by 15
  11. 2021HyperSPNs: Compact and Expressive Probabilistic Circuitscited by 14
  12. 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
  13. 2024Banishing LLM Hallucinations Requires Rethinking Generalizationcited by 13
  14. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 12
  15. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 12
  16. 2021Enabling Binary Neural Network Training on the Edgecited by 11
  17. 2021Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memorycited by 11
  18. 2021Efficient Scaling of Dynamic Graph Neural Networkscited by 11
  19. 2023Score-based Data Assimilation for a Two-Layer Quasi-Geostrophic Modelcited by 11
  20. 2022Recurrent Neural Networks for Dynamical Systems: Applications to Ordinary Differential Equations, Collective Motion, and Hydrological Modelingcited by 9
  21. 2022DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Trainingcited by 9
  22. 2024Learnable Sparsity for Vision Generative Modelscited by 9
  23. 2022p-Meta: Towards On-device Deep Model Adaptationcited by 8
  24. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  25. 2024FREE: Faster and Better Data-Free Meta-Learningcited by 8
  26. 2023Long-term Dependency for 3D Reconstruction of Freehand Ultrasound Without External Trackercited by 7
  27. 2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
  28. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  29. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  30. 2023Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Modelscited by 6
  31. 2024Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reductioncited by 5
  32. 2024On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollerscited by 5
  33. 2025Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizercited by 5
  34. 2025From Tiny Machine Learning to Tiny Deep Learning: A Surveycited by 5
  35. 2022Survey on Large Scale Neural Network Trainingcited by 4
  36. 2022DIVISION: Memory Efficient Training via Dual Activation Precisioncited by 4
  37. 2024Block Selective Reprogramming for On-device Training of Vision Transformerscited by 4
  38. 2024Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edgecited by 4

50 most cited of 78 All 78 (JSON)