# Efficient Memory Management for Deep Neural Net Inference

arXiv: https://arxiv.org/abs/2001.03288 · HTML: https://mlsanity.com/paper/2001.03288/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2020.

## Built on by (most cited first, 12 of 12)

- [Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations](https://mlsanity.com/paper/2304.11267/) (2023)
- [PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management](https://mlsanity.com/paper/2108.05818/) (2021)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Demand Layering for Real-Time DNN Inference with Minimized Memory Usage](https://mlsanity.com/paper/2210.04024/) (2022)
- [SoD2: Statically Optimizing Dynamic Deep Neural Network](https://mlsanity.com/paper/2403.00176/) (2024)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [Scaling On-Device GPU Inference for Large Generative Models](https://mlsanity.com/paper/2505.00232/) (2025)
- [Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Taming](https://mlsanity.com/paper/2601.06562/) (2026)
- [ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout](https://mlsanity.com/paper/2310.19295/) (2023)
- [Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design](https://mlsanity.com/paper/2407.02813/) (2024)
- [Input Resolution Downsizing as a Compression Technique for Vision Deep Learning Systems](https://mlsanity.com/paper/2504.03749/) (2025)
- [Futureproof Static Memory Planning](https://mlsanity.com/paper/2504.04874/) (2025)

## Abstract (arXiv)

While deep neural net inference was considered a task for servers only, latest advances in technology allow the task of inference to be moved to mobile and embedded devices, desired for various reasons ranging from latency to privacy. These devices are not only limited by their compute power and battery, but also by their inferior physical memory and cache, and thus, an efficient memory manager becomes a crucial component for deep neural net inference at the edge. We explore various strategies to smartly share memory buffers among intermediate tensors in deep neural nets. Employing these can result in up to 11% smaller memory footprint than the state of the art.