# Efficient Memory Management for GPU-based Deep Learning Systems

arXiv: https://arxiv.org/abs/1903.06631 · HTML: https://mlsanity.com/paper/1903.06631/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2019.

## Built on by (most cited first, 6 of 6)

- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent](https://mlsanity.com/paper/2303.02868/) (2023)
- [Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers](https://mlsanity.com/paper/2202.01306/) (2022)
- [Survey on Large Scale Neural Network Training](https://mlsanity.com/paper/2202.10435/) (2022)
- [TENSILE: A Tensor granularity dynamic GPU memory scheduling method toward multiple dynamic workloads system](https://mlsanity.com/paper/2105.13336/) (2021)
- [Analysis of memory consumption by neural networks based on hyperparameters](https://mlsanity.com/paper/2110.11424/) (2021)

## Abstract (arXiv)

GPU (graphics processing unit) has been used for many data-intensive applications. Among them, deep learning systems are one of the most important consumer systems for GPU nowadays. As deep learning applications impose deeper and larger models in order to achieve higher accuracy, memory management becomes an important research topic for deep learning systems, given that GPU has limited memory size. Many approaches have been proposed towards this issue, e.g., model compression and memory swapping. However, they either degrade the model accuracy or require a lot of manual intervention. In this paper, we propose two orthogonal approaches to reduce the memory cost from the system perspective. Our approaches are transparent to the models, and thus do not affect the model accuracy. They are achieved by exploiting the iterative nature of the training algorithm of deep learning to derive the lifetime and read/write order of all variables. With the lifetime semantics, we are able to implement a memory pool with minimal fragments. However, the optimization problem is NP-complete. We propose a heuristic algorithm that reduces up to 13.3% of memory compared with Nvidia's default memory pool with equal time complexity. With the read/write semantics, the variables that are not in use can be swapped out from GPU to CPU to reduce the memory footprint. We propose multiple swapping strategies to automatically decide which variable to swap and when to swap out (in), which reduces the memory cost by up to 34.2% without communication overhead.