# Profile-guided memory optimization for deep neural networks

arXiv: https://arxiv.org/abs/1804.10001 · HTML: https://mlsanity.com/paper/1804.10001/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2018.

## Built on by (most cited first, 6 of 6)

- [Demand Layering for Real-Time DNN Inference with Minimized Memory Usage](https://mlsanity.com/paper/2210.04024/) (2022)
- [OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks](https://mlsanity.com/paper/2210.12924/) (2022)
- [MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training](https://mlsanity.com/paper/2407.12117/) (2024)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout](https://mlsanity.com/paper/2310.19295/) (2023)
- [Futureproof Static Memory Planning](https://mlsanity.com/paper/2504.04874/) (2025)

## Abstract (arXiv)

Recent years have seen deep neural networks (DNNs) becoming wider and deeper to achieve better performance in many applications of AI. Such DNNs however require huge amounts of memory to store weights and intermediate results (e.g., activations, feature maps, etc.) in propagation. This requirement makes it difficult to run the DNNs on devices with limited, hard-to-extend memory, degrades the running time performance, and restricts the design of network models. We address this challenge by developing a novel profile-guided memory optimization to efficiently and quickly allocate memory blocks during the propagation in DNNs. The optimization utilizes a simple and fast heuristic algorithm based on the two-dimensional rectangle packing problem. Experimenting with well-known neural network models, we confirm that our method not only reduces the memory consumption by up to 49.5% but also accelerates training and inference by up to a factor of four thanks to the rapidity of the memory allocation and the ability to use larger mini-batch sizes.