mlsanityData to 7 Oct 2026

ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout

arXiv 2310.19295 ↗First posted 30 Oct 2023cs.LG cs.AI cs.DBcited by 0

Not on the map. Why?

Abstract (arXiv)

As deep learning models continue to increase in size, the memory requirements for training have surged. While high-level techniques like offloading, recomputation, and compression can alleviate memory pressure, they also introduce overheads. However, a memory-efficient execution plan that includes a reasonable operator execution order and tensor memory layout can significantly increase the models' memory efficiency and reduce overheads from high-level techniques. In this paper, we propose ROAM which operates on computation graph level to derive memory-efficient execution plan with optimized operator order and tensor memory layout for models. We first propose sophisticated theories that carefully consider model structure and training memory load to support optimization for large complex graphs that have not been well supported in the past. An efficient tree-based algorithm is further proposed to search task divisions automatically, along with delivering high performance and effectiveness to solve the problem. Experiments show that ROAM achieves a substantial memory reduction of 35.7%, 13.3%, and 27.2% compared to Pytorch and two state-of-the-art methods and offers a remarkable 53.7x speedup. The evaluation conducted on the expansive GPT2-XL further validates ROAM's scalability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

33 of 56 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2016An overview of gradient descent optimization algorithmscited by 1,021
  12. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
21 more
  1. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  3. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  4. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  5. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  6. 2022On-Device Training Under 256KB Memorycited by 158
  7. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 77
  8. 2019Low-Memory Neural Network Training: A Technical Reportcited by 56
  9. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  10. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  11. 2020Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devicescited by 27
  12. 2020ProGraML: Graph-based Deep Learning for Program Optimization and Analysiscited by 23
  13. 2020Memory Optimization for Deep Networkscited by 15
  14. 2020Efficient Memory Management for Deep Neural Net Inferencecited by 12
  15. 2021COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compressioncited by 8
  16. 2018Profile-guided memory optimization for deep neural networkscited by 6
  17. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  18. 2022Memory Planning for Deep Neural Networkscited by 3
  19. 2023TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networkscited by 3
  20. 2016vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Designcited by 1
  21. 2018Optimal Gradient Checkpoint Search for Arbitrary Computation Graphscited by 1

Built on by

Not cited yet.