mlsanityData to 7 Oct 2026

Training Large Neural Networks with Constant Memory using a New Execution Algorithm

arXiv 2002.05645 ↗First posted 13 Feb 2020cs.LG cs.DC stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

Widely popular transformer-based NLP models such as BERT and Turing-NLG have enormous capacity trending to billions of parameters. Current execution methods demand brute-force resources such as HBM devices and high speed interconnectivity for data parallelism. In this paper, we introduce a new relay-style execution technique called L2L (layer-to-layer) where at any given moment, the device memory is primarily populated only with the executing layer(s)'s footprint. The model resides in the DRAM memory attached to either a CPU or an FPGA as an entity we call eager param-server (EPS). To overcome the bandwidth issues of shuttling parameters to and from EPS, the model is executed a layer at a time across many micro-batches instead of the conventional method of minibatches over whole model. L2L is implemented using 16GB V100 devices for BERT-Large running it with a device batch size of up to 256. Our results show 45% reduction in memory and 40% increase in the throughput compared to the state-of-the-art baseline. L2L is also able to fit models up to 50 Billion parameters on a machine with a single 16GB V100 and 512GB CPU memory and without requiring any model partitioning. L2L scales to arbitrary depth allowing researchers to develop on affordable devices which is a big step toward democratizing AI. By running the optimizer in the host EPS, we show a new form of mixed precision for faster throughput and convergence. In addition, the EPS enables dynamic neural architecture approaches by varying layers across iterations. Finally, we also propose and demonstrate a constant memory variation of L2L and we propose future enhancements. This work has been performed on GPUs first, but also targeted towards all high TFLOPS/Watt accelerators.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 8; 2024: 5; 2025: 8; 2026: 3.

34 papers.

Sort:

By area:

  1. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  3. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  4. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  5. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  6. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  7. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  8. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  9. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  10. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 14
  11. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  12. 2025GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Developmentcited by 10
22 more
  1. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  2. 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
  3. 2024Xiwu: A Basis Flexible and Learnable LLM for High Energy Physicscited by 7
  4. 2022Instilling Type Knowledge in Language Models via Multi-Task QAcited by 5
  5. 2023Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Trainingcited by 5
  6. 2022Survey on Large Scale Neural Network Trainingcited by 4
  7. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  8. 2021Hydra: A System for Large Multi-Model Deep Learningcited by 3
  9. 2025Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storagecited by 3
  10. 2022DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputationcited by 2
  11. 2022Elixir: Train a Large Language Model on a Small GPU Clustercited by 2
  12. 2023Vocabulary-level Memory Efficiency for Language Model Fine-tuningcited by 2
  13. 202510Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Trainingcited by 2
  14. 2026FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Trainingcited by 2
  15. 2022Benchmark Assessment for DeepSpeed Optimization Librarycited by 1
  16. 2025Merging Feed-Forward Sublayers for Compressed Transformerscited by 1
  17. 2024LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reductioncited by 0
  18. 2025GPU Memory Usage Optimization for Backward Propagation in Deep Network Trainingcited by 0
  19. 2025Symbiosis: Multi-Adapter Inference and Fine-Tuningcited by 0
  20. 2025Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Modelscited by 0
  21. 2026Efficient Software Vulnerability Detection Using Transformer-based Modelscited by 0
  22. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0