# G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations

arXiv: https://arxiv.org/abs/2310.09443 · HTML: https://mlsanity.com/paper/2310.09443/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 71 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning](https://mlsanity.com/paper/1908.03072/) (2019)
- [Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect](https://mlsanity.com/paper/1903.04611/) (2019)
- [Bandana: Using Non-volatile Memory for Storing Deep Learning Models](https://mlsanity.com/paper/1811.05922/) (2018)
- [TFLMS: Large Model Support in TensorFlow by Graph Rewriting](https://mlsanity.com/paper/1807.02037/) (2018)
- [Analyzing Machine Learning Workloads Using a Detailed GPU Simulator](https://mlsanity.com/paper/1811.08933/) (2018)
- [Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning](https://mlsanity.com/paper/1902.06468/) (2019)
- [NeuMMU: Architectural Support for Efficient Address Translations in Neural Processing Units](https://mlsanity.com/paper/1911.06859/) (2019)
- [ZnG: Architecting GPU Multi-Processors with New Flash for Scalable Data Analysis](https://mlsanity.com/paper/2006.08975/) (2020)

## Built on by (most cited first, 10 of 10)

- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [APOLLO: SGD-like Memory, AdamW-level Performance](https://mlsanity.com/paper/2412.05270/) (2024)
- [Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?](https://mlsanity.com/paper/2410.01623/) (2024)
- [Training and Serving System of Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2401.02643/) (2024)
- [MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving](https://mlsanity.com/paper/2510.14557/) (2025)
- [Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage](https://mlsanity.com/paper/2506.06472/) (2025)
- [Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms](https://mlsanity.com/paper/2408.04104/) (2024)
- [Practical offloading for fine-tuning LLM on commodity GPU via learned sparse projectors](https://mlsanity.com/paper/2406.10181/) (2024)
- [ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques](https://mlsanity.com/paper/2507.11506/) (2025)
- [Efficient Training on Multiple Consumer GPUs with RoundPipe](https://mlsanity.com/paper/2604.27085/) (2026)

## Abstract (arXiv)

To break the GPU memory wall for scaling deep learning workloads, a variety of architecture and system techniques have been proposed recently. Their typical approaches include memory extension with flash memory and direct storage access. However, these techniques still suffer from suboptimal performance and introduce complexity to the GPU memory management, making them hard to meet the scalability requirement of deep learning workloads today. In this paper, we present a unified GPU memory and storage architecture named G10 driven by the fact that the tensor behaviors of deep learning workloads are highly predictable. G10 integrates the host memory, GPU memory, and flash memory into a unified memory space, to scale the GPU memory capacity while enabling transparent data migrations. Based on this unified GPU memory and storage architecture, G10 utilizes compiler techniques to characterize the tensor behaviors in deep learning workloads. Therefore, it can schedule data migrations in advance by considering the available bandwidth of flash memory and host memory. The cooperative mechanism between deep learning compilers and the unified memory architecture enables G10 to hide data transfer overheads in a transparent manner. We implement G10 based on an open-source GPU simulator. Our experiments demonstrate that G10 outperforms state-of-the-art GPU memory solutions by up to 1.75×, without code modifications to deep learning workloads. With the smart data migration mechanism, G10 can reach 90.3% of the performance of the ideal case assuming unlimited GPU memory.