# Training Large Neural Networks with Constant Memory using a New Execution Algorithm

arXiv: https://arxiv.org/abs/2002.05645 · HTML: https://mlsanity.com/paper/2002.05645/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 30 of 34)

- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)
- [Petals: Collaborative Inference and Fine-tuning of Large Models](https://mlsanity.com/paper/2209.01188/) (2022)
- [SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient](https://mlsanity.com/paper/2301.11913/) (2023)
- [SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices](https://mlsanity.com/paper/2406.02532/) (2024)
- [Distributed Inference and Fine-tuning of Large Language Models Over The Internet](https://mlsanity.com/paper/2312.08361/) (2023)
- [PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management](https://mlsanity.com/paper/2108.05818/) (2021)
- [Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System](https://mlsanity.com/paper/2403.06664/) (2024)
- [HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy](https://mlsanity.com/paper/2401.15207/) (2024)
- [GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Development](https://mlsanity.com/paper/2503.00686/) (2025)
- [SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs](https://mlsanity.com/paper/2505.17052/) (2025)
- [Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models](https://mlsanity.com/paper/2312.14751/) (2023)
- [Xiwu: A Basis Flexible and Learnable LLM for High Energy Physics](https://mlsanity.com/paper/2404.08001/) (2024)
- [Instilling Type Knowledge in Language Models via Multi-Task QA](https://mlsanity.com/paper/2204.13796/) (2022)
- [Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Training](https://mlsanity.com/paper/2305.19982/) (2023)
- [Survey on Large Scale Neural Network Training](https://mlsanity.com/paper/2202.10435/) (2022)
- [Systems for Parallel and Distributed Large-Model Deep Learning Training](https://mlsanity.com/paper/2301.02691/) (2023)
- [Hydra: A System for Large Multi-Model Deep Learning](https://mlsanity.com/paper/2110.08633/) (2021)
- [Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage](https://mlsanity.com/paper/2506.06472/) (2025)
- [DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation](https://mlsanity.com/paper/2203.15980/) (2022)
- [Elixir: Train a Large Language Model on a Small GPU Cluster](https://mlsanity.com/paper/2212.05339/) (2022)
- [Vocabulary-level Memory Efficiency for Language Model Fine-tuning](https://mlsanity.com/paper/2309.08708/) (2023)
- [10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training](https://mlsanity.com/paper/2511.14124/) (2025)
- [FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training](https://mlsanity.com/paper/2606.22932/) (2026)
- [Benchmark Assessment for DeepSpeed Optimization Library](https://mlsanity.com/paper/2202.12831/) (2022)
- [Merging Feed-Forward Sublayers for Compressed Transformers](https://mlsanity.com/paper/2501.06126/) (2025)
- [LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction](https://mlsanity.com/paper/2401.11471/) (2024)
- [GPU Memory Usage Optimization for Backward Propagation in Deep Network Training](https://mlsanity.com/paper/2502.12499/) (2025)

## Abstract (arXiv)

Widely popular transformer-based NLP models such as BERT and Turing-NLG have enormous capacity trending to billions of parameters. Current execution methods demand brute-force resources such as HBM devices and high speed interconnectivity for data parallelism. In this paper, we introduce a new relay-style execution technique called L2L (layer-to-layer) where at any given moment, the device memory is primarily populated only with the executing layer(s)'s footprint. The model resides in the DRAM memory attached to either a CPU or an FPGA as an entity we call eager param-server (EPS). To overcome the bandwidth issues of shuttling parameters to and from EPS, the model is executed a layer at a time across many micro-batches instead of the conventional method of minibatches over whole model. L2L is implemented using 16GB V100 devices for BERT-Large running it with a device batch size of up to 256. Our results show 45% reduction in memory and 40% increase in the throughput compared to the state-of-the-art baseline. L2L is also able to fit models up to 50 Billion parameters on a machine with a single 16GB V100 and 512GB CPU memory and without requiring any model partitioning. L2L scales to arbitrary depth allowing researchers to develop on affordable devices which is a big step toward democratizing AI. By running the optimizer in the host EPS, we show a new form of mixed precision for faster throughput and convergence. In addition, the EPS enables dynamic neural architecture approaches by varying layers across iterations. Finally, we also propose and demonstrate a constant memory variation of L2L and we propose future enhancements. This work has been performed on GPUs first, but also targeted towards all high TFLOPS/Watt accelerators.