# Low-Memory Neural Network Training: A Technical Report

arXiv: https://arxiv.org/abs/1904.10631 · HTML: https://mlsanity.com/paper/1904.10631/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 30 of 57)

- [ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models](https://mlsanity.com/paper/2310.10505/) (2023)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing](https://mlsanity.com/paper/2209.14635/) (2022)
- [Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models](https://mlsanity.com/paper/2405.15574/) (2024)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge](https://mlsanity.com/paper/2307.09988/) (2023)
- [Phantom of Latent for Large Language and Vision Models](https://mlsanity.com/paper/2409.14713/) (2024)
- [TroL: Traversal of Layers for Large Language and Vision Models](https://mlsanity.com/paper/2406.12246/) (2024)
- [Improving Formality Style Transfer with Context-Aware Rule Injection](https://mlsanity.com/paper/2106.00210/) (2021)
- [Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challenges](https://mlsanity.com/paper/2203.12852/) (2022)
- [Enabling Binary Neural Network Training on the Edge](https://mlsanity.com/paper/2102.04270/) (2021)
- [LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms](https://mlsanity.com/paper/2311.11420/) (2023)
- [Fine-tuning Whisper on Low-Resource Languages for Real-World Applications](https://mlsanity.com/paper/2412.15726/) (2024)
- [On using distributed representations of source code for the detection of C security vulnerabilities](https://mlsanity.com/paper/2106.01367/) (2021)
- [VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models](https://mlsanity.com/paper/2412.01822/) (2024)
- [GenRecal: Generation after Recalibration from Large to Small Vision-Language Models](https://mlsanity.com/paper/2506.15681/) (2025)
- [DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training](https://mlsanity.com/paper/2202.13808/) (2022)
- [Parameterized Hypercomplex Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/2103.16584/) (2021)
- [How to guess a gradient](https://mlsanity.com/paper/2312.04709/) (2023)
- [Seeing World Dynamics in a Nutshell](https://mlsanity.com/paper/2502.03465/) (2025)
- [Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2505.20789/) (2025)
- [An improved LogNNet classifier for IoT application](https://mlsanity.com/paper/2105.14412/) (2021)
- [Memory-adaptive Depth-wise Heterogeneous Federated Learning](https://mlsanity.com/paper/2303.04887/) (2023)
- [A Study of Optimizations for Fine-tuning Large Language Models](https://mlsanity.com/paper/2406.02290/) (2024)
- [MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.03010/) (2022)
- [Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Training](https://mlsanity.com/paper/2305.19982/) (2023)
- [Enabling On-Device Smartphone GPU based Training: Lessons Learned](https://mlsanity.com/paper/2202.10100/) (2022)
- [Loop Unrolled Shallow Equilibrium Regularizer (LUSER) -- A Memory-Efficient Inverse Problem Solver](https://mlsanity.com/paper/2210.04987/) (2022)
- [Systems for Parallel and Distributed Large-Model Deep Learning Training](https://mlsanity.com/paper/2301.02691/) (2023)

## Abstract (arXiv)

Memory is increasingly often the bottleneck when training neural network models. Despite this, techniques to lower the overall memory requirements of training have been less widely studied compared to the extensive literature on reducing the memory requirements of inference. In this paper we study a fundamental question: How much memory is actually needed to train a neural network? To answer this question, we profile the overall memory usage of training on two representative deep learning benchmarks -- the WideResNet model for image classification and the DynamicConv Transformer model for machine translation -- and comprehensively evaluate four standard techniques for reducing the training memory requirements: (1) imposing sparsity on the model, (2) using low precision, (3) microbatching, and (4) gradient checkpointing. We explore how each of these techniques in isolation affects both the peak memory usage of training and the quality of the end model, and explore the memory, accuracy, and computation tradeoffs incurred when combining these techniques. Using appropriate combinations of these techniques, we show that it is possible to the reduce the memory required to train a WideResNet-28-2 on CIFAR-10 by up to 60.7x with a 0.4% loss in accuracy, and reduce the memory required to train a DynamicConv model on IWSLT'14 German to English translation by up to 8.7x with a BLEU score drop of 0.15.