# Toward Robust and Efficient ML-Based GPU Caching for Modern Inference

arXiv: https://arxiv.org/abs/2509.20979 · HTML: https://mlsanity.com/paper/2509.20979/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [Competitive caching with machine learned advice](https://mlsanity.com/paper/1802.05399/) (2018)
- [Algorithms with Predictions](https://mlsanity.com/paper/2006.09123/) (2020)
- [Near-Optimal Bounds for Online Caching with Machine Learned Advice](https://mlsanity.com/paper/1910.12172/) (2019)
- [RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing](https://mlsanity.com/paper/1912.12953/) (2019)
- [TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning](https://mlsanity.com/paper/1908.03072/) (2019)
- [KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider](https://mlsanity.com/paper/2506.02634/) (2025)
- [Better and Simpler Learning-Augmented Online Caching](https://mlsanity.com/paper/2005.13716/) (2020)
- [Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference](https://mlsanity.com/paper/2503.08311/) (2025)
- [Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems](https://mlsanity.com/paper/2210.10629/) (2022)
- [An Imitation Learning Approach for Cache Replacement](https://mlsanity.com/paper/2006.16239/) (2020)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [Bandana: Using Non-volatile Memory for Storing Deep Learning Models](https://mlsanity.com/paper/1811.05922/) (2018)
- [Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression](https://mlsanity.com/paper/2505.06901/) (2025)
- [AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure](https://mlsanity.com/paper/2504.03648/) (2025)
- [Algorithms for Caching and MTS with reduced number of predictions](https://mlsanity.com/paper/2404.06280/) (2024)
- [A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models](https://mlsanity.com/paper/2210.08804/) (2022)
- [UpDLRM: Accelerating Personalized Recommendation using Real-World PIM Architecture](https://mlsanity.com/paper/2406.13941/) (2024)
- [A Dynamic Hash Table for the GPU](https://mlsanity.com/paper/1710.11246/) (2017)
- [A Learned Cache Eviction Framework with Minimal Overhead](https://mlsanity.com/paper/2301.11886/) (2023)
- [Robustifying Learning-Augmented Caching Efficiently without Compromising 1-Consistency](https://mlsanity.com/paper/2507.16242/) (2025)

## Built on by (most cited first, 3 of 3)

- [SCION: Size-aware Policy Orchestration for Nonstationary Object Caches (Long Paper Version)](https://mlsanity.com/paper/2605.01055/) (2026)
- [When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving](https://mlsanity.com/paper/2605.04450/) (2026)
- [Memory as a Wasting Asset: Pricing Flash Endurance for Embodied Agents, and the Limits of Doing So](https://mlsanity.com/paper/2606.18144/) (2026)

## Abstract (arXiv)

In modern GPU inference, cache efficiency remains a major bottleneck, and heuristic policies such as textscLRU can perform far worse than the offline optimum. Existing learning-based caching systems improve hit rates mainly through predictor design, but often follow learned predictions blindly, making performance unreliable when predictions are inaccurate. In contrast, emerging learning-augmented caching algorithms~citepmlr-v80-lykouris18a,mitzenmacher2022algorithms provide performance guarantees by carefully integrating predictions into caching policies, achieving both consistency (near-optimality under perfect predictions) and robustness (bounded worst-case performance under prediction errors). However, deployment remains challenging. A practical algorithm should satisfy strict time and space efficiency constraints, which some theoretical work overlooks, while also incurring low deployment overhead. We propose learning-augmented LRU, a deployment-oriented learning-augmented caching algorithm that guarantees 1-consistency and O(k)-robustness, incurs low time and space overhead, and maintains strong compatibility. We further build a GPU cache, called textscLCR, on top of learning-augmented LRU to benefit from its theoretical guarantees and translate them into practical performance. In experiments, textscLCR reduces P99 time-to-first-token (TTFT) by up to 28.3% on LLM workloads and increases throughput by up to 24.2% on deep learning recommendation (DLRM) workloads. Even with poor predictions, performance degrades gracefully and remains close to textscLRU, demonstrating robustness with practical value.