# Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning

arXiv: https://arxiv.org/abs/1902.06468 · HTML: https://mlsanity.com/paper/1902.06468/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2019.

## Built on by (most cited first, 9 of 9)

- [SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architectures](https://mlsanity.com/paper/2205.04711/) (2022)
- [Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards](https://mlsanity.com/paper/2205.04702/) (2022)
- [G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations](https://mlsanity.com/paper/2310.09443/) (2023)
- [Ascend HiFloat8 Format for Deep Learning](https://mlsanity.com/paper/2409.16626/) (2024)
- [GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networks](https://mlsanity.com/paper/2203.00158/) (2022)
- [DiVa: An Accelerator for Differentially Private Machine Learning](https://mlsanity.com/paper/2208.12392/) (2022)
- [Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage](https://mlsanity.com/paper/2506.06472/) (2025)
- [10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training](https://mlsanity.com/paper/2511.14124/) (2025)
- [NEON: Enabling Efficient Support for Nonlinear Operations in Resistive RAM-based Neural Network Accelerators](https://mlsanity.com/paper/2211.05730/) (2022)

## Abstract (arXiv)

As the models and the datasets to train deep learning (DL) models scale, system architects are faced with new challenges, one of which is the memory capacity bottleneck, where the limited physical memory inside the accelerator device constrains the algorithm that can be studied. We propose a memory-centric deep learning system that can transparently expand the memory capacity available to the accelerators while also providing fast inter-device communication for parallel training. Our proposal aggregates a pool of memory modules locally within the device-side interconnect, which are decoupled from the host interface and function as a vehicle for transparent memory capacity expansion. Compared to conventional systems, our proposal achieves an average 2.8x speedup on eight DL applications and increases the system-wide memory capacity to tens of TBs.