# GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching

arXiv: https://arxiv.org/abs/2401.08156 · HTML: https://mlsanity.com/paper/2401.08156/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2024.

## Built on by (most cited first, 15 of 15)

- [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention](https://mlsanity.com/paper/2405.04437/) (2024)
- [Jenga: Effective Memory Management for Serving LLM with Heterogeneity](https://mlsanity.com/paper/2503.18292/) (2025)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type](https://mlsanity.com/paper/2502.18755/) (2025)
- [vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving](https://mlsanity.com/paper/2407.15309/) (2024)
- [M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization](https://mlsanity.com/paper/2601.19213/) (2026)
- [Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization](https://mlsanity.com/paper/2503.19050/) (2025)
- [Clinical Insights: A Comprehensive Review of Language Models in Medicine](https://mlsanity.com/paper/2408.11735/) (2024)
- [MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training](https://mlsanity.com/paper/2407.12117/) (2024)
- [Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs](https://mlsanity.com/paper/2512.20210/) (2025)
- [HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism](https://mlsanity.com/paper/2507.00394/) (2025)
- [STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planning](https://mlsanity.com/paper/2507.16274/) (2025)
- [xGR: Efficient Generative Recommendation Serving at Scale](https://mlsanity.com/paper/2512.11529/) (2025)
- [MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems](https://mlsanity.com/paper/2605.05888/) (2026)
- [Dynamic HBM Repartitioning for Multi-Turn MoE Serving](https://mlsanity.com/paper/2609.13537/) (2026)

## Abstract (arXiv)

Large-scale deep neural networks (DNNs), such as large language models (LLMs), have revolutionized the artificial intelligence (AI) field and become increasingly popular. However, training or fine-tuning such models requires substantial computational power and resources, where the memory capacity of a single acceleration device like a GPU is one of the most important bottlenecks. Owing to the prohibitively large overhead (e.g., 10 ×) of GPUs' native memory allocator, DNN frameworks like PyTorch and TensorFlow adopt a caching allocator that maintains a memory pool with a splitting mechanism for fast memory (de)allocation. Unfortunately, the caching allocator's efficiency degrades quickly for popular memory reduction techniques such as recomputation, offloading, distributed training, and low-rank adaptation. The primary reason is that those memory reduction techniques introduce frequent and irregular memory (de)allocation requests, leading to severe fragmentation problems for the splitting-based caching allocator. To mitigate this fragmentation problem, we propose a novel memory allocation framework based on low-level GPU virtual memory management called GPU memory lake (GMLake). GMLake employs a novel virtual memory stitching (VMS) mechanism, which can fuse or combine non-contiguous memory blocks with a virtual memory address mapping. GMLake can reduce an average of 9.2 GB (up to 25 GB) GPU memory usage and 15% (up to 33% ) fragmentation among eight LLM models on GPU A100 with 80 GB memory. GMLake is completely transparent to the DNN models and memory reduction techniques and ensures the seamless execution of resource-intensive deep-learning tasks. We have open-sourced GMLake at https://github.com/intelligent-machine-learning/glake/tree/main/GMLake.