# MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches

arXiv: https://arxiv.org/abs/2604.22881 · HTML: https://mlsanity.com/paper/2604.22881/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1803.02349/) (2018)
- [HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling](https://mlsanity.com/paper/2409.12740/) (2024)
- [MTGR: Industrial-Scale Generative Recommendation Framework in Meituan](https://mlsanity.com/paper/2505.18654/) (2025)
- [OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender](https://mlsanity.com/paper/2510.26104/) (2025)
- [Generative Recommender with End-to-End Learnable Item Tokenization](https://mlsanity.com/paper/2409.05546/) (2024)
- [CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendation](https://mlsanity.com/paper/2405.02429/) (2024)
- [OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System](https://mlsanity.com/paper/2509.18091/) (2025)
- [Inductive Generative Recommendation via Retrieval-based Speculation](https://mlsanity.com/paper/2410.02939/) (2024)
- [Bridging Search and Recommendation in Generative Retrieval: Does One Task Help the Other?](https://mlsanity.com/paper/2410.16823/) (2024)
- [Preference Discerning with LLM-Enhanced Generative Retrieval](https://mlsanity.com/paper/2412.08604/) (2024)
- [ULMRec: User-centric Large Language Model for Sequential Recommendation](https://mlsanity.com/paper/2412.05543/) (2024)
- [Token-level Collaborative Alignment for LLM-based Generative Recommendation](https://mlsanity.com/paper/2601.18457/) (2026)

## Built on by (most cited first, 2 of 2)

- [Enabling High-Bandwidth Flash for Generative Recommendation Serving with Write-Aware KV Cache Policy](https://mlsanity.com/paper/2609.07175/) (2026)
- [Self-Evolving Memory for Generative Recommendation](https://mlsanity.com/paper/2609.15598/) (2026)

## Abstract (arXiv)

Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%).