# 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training

arXiv: https://arxiv.org/abs/2511.14124 · HTML: https://mlsanity.com/paper/2511.14124/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 18 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [Training Large Neural Networks with Constant Memory using a New Execution Algorithm](https://mlsanity.com/paper/2002.05645/) (2020)
- [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://mlsanity.com/paper/2502.09334/) (2025)
- [FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models](https://mlsanity.com/paper/2501.10714/) (2025)
- [Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System](https://mlsanity.com/paper/2403.06664/) (2024)
- [Seesaw: High-throughput LLM Inference via Model Re-sharding](https://mlsanity.com/paper/2503.06433/) (2025)
- [Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs](https://mlsanity.com/paper/1903.02596/) (2019)
- [Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning](https://mlsanity.com/paper/1902.06468/) (2019)
- [DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimization](https://mlsanity.com/paper/2307.04963/) (2023)
- [SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips](https://mlsanity.com/paper/2509.21271/) (2025)
- [ZenFlow: Enabling Stall-Free Offloading for LLM Training](https://mlsanity.com/paper/2505.12242/) (2025)
- [Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores](https://mlsanity.com/paper/2503.10725/) (2025)
- [MemAscend: System Memory Optimization for SSD-Offloaded LLM Fine-Tuning](https://mlsanity.com/paper/2505.23254/) (2025)

## Built on by (most cited first, 2 of 2)

- [LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training](https://mlsanity.com/paper/2608.11919/) (2026)
- [Horizon-LM: A RAM-Centric Architecture for LLM Training](https://mlsanity.com/paper/2602.04816/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) in the cloud faces growing memory bottlenecks due to the limited capacity and high cost of GPUs. While GPU memory offloading to CPU and NVMe has made large-scale training more feasible, existing approaches suffer from high tensor migration latency and suboptimal device memory utilization, ultimately increasing training time and cloud costs. To address these challenges, we present 10Cache, a resource-aware tensor caching and migration system that accelerates LLM training by intelligently coordinating memory usage across GPU, CPU, and NVMe tiers. 10Cache profiles tensor execution order to construct prefetch policies, allocates memory buffers in pinned memory based on tensor size distributions, and reuses memory buffers to minimize allocation overhead. Designed for cloud-scale deployments, 10Cache improves memory efficiency and reduces reliance on high-end GPUs. Across diverse LLM workloads, it achieves up to 2x speedup in training time, improves GPU cache hit rate by up to 86.6x, and increases CPU/GPU memory utilization by up to 2.15x and 1.33x, respectively, compared to state-of-the-art offloading methods. These results demonstrate that 10Cache is a practical and scalable solution for optimizing LLM training throughput and resource efficiency in cloud environments.