# RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference

arXiv: https://arxiv.org/abs/2601.01712 · HTML: https://mlsanity.com/paper/2601.01712/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 30 of 33)

- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09433/) (2019)
- [ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition](https://mlsanity.com/paper/2402.15220/) (2024)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling](https://mlsanity.com/paper/2409.12740/) (2024)
- [OneRec-Think: In-Text Reasoning for Generative Recommendation](https://mlsanity.com/paper/2510.11639/) (2025)
- [MTGR: Industrial-Scale Generative Recommendation Framework in Meituan](https://mlsanity.com/paper/2505.18654/) (2025)
- [Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time](https://mlsanity.com/paper/1711.07601/) (2017)
- [End-to-End User Behavior Retrieval in Click-Through RatePrediction Model](https://mlsanity.com/paper/2108.04468/) (2021)
- [RankMixer: Scaling Up Ranking Models in Industrial Recommenders](https://mlsanity.com/paper/2507.15551/) (2025)
- [OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender](https://mlsanity.com/paper/2510.26104/) (2025)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [A Survey on User Behavior Modeling in Recommender Systems](https://mlsanity.com/paper/2302.11087/) (2023)
- [Serving Large Language Models on Huawei CloudMatrix384](https://mlsanity.com/paper/2506.12708/) (2025)
- [Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving](https://mlsanity.com/paper/2508.01989/) (2025)
- [A Comprehensive Survey on Retrieval Methods in Recommender Systems](https://mlsanity.com/paper/2407.21022/) (2024)
- [LLMs for User Interest Exploration in Large-scale Recommendation Systems](https://mlsanity.com/paper/2405.16363/) (2024)
- [Unlocking Scaling Law in Industrial Recommendation Systems with a Three-step Paradigm based Large User Model](https://mlsanity.com/paper/2502.08309/) (2025)
- [Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation](https://mlsanity.com/paper/2503.20552/) (2025)

## Built on by (most cited first, 2 of 2)

- [When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving](https://mlsanity.com/paper/2605.04450/) (2026)
- [SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching](https://mlsanity.com/paper/2609.08443/) (2026)

## Abstract (arXiv)

Real-time recommender systems execute multi-stage cascades (retrieval, pre-processing, fine-grained ranking) under strict tail-latency SLOs, leaving only tens of milliseconds for ranking. Generative recommendation (GR) models can improve quality by consuming long user-behavior sequences, but in production their online sequence length is tightly capped by the ranking-stage P99 budget. We observe that the majority of GR tokens encode user behaviors that are independent of the item candidates, suggesting an opportunity to pre-infer a user-behavior prefix once and reuse it during ranking rather than recomputing it on the critical path. Realizing this idea at industrial scale is non-trivial: the prefix cache must survive across multiple pipeline stages before the final ranking instance is determined, the user population implies cache footprints far beyond a single device, and indiscriminate pre-inference would overload shared resources under high QPS. We present RelayGR, a production system that enables in-HBM relay-race inference for GR. RelayGR selectively pre-infers long-term user prefixes, keeps their KV caches resident in HBM over the request lifecycle, and ensures the subsequent ranking can consume them without remote fetches. RelayGR combines three techniques: 1) a sequence-aware trigger that admits only at-risk requests under a bounded cache footprint and pre-inference load, 2) an affinity-aware router that co-locates cache production and consumption by routing both the auxiliary pre-infer signal and the ranking request to the same instance, and 3) a memory-aware expander that uses server-local DRAM to capture short-term cross-request reuse while avoiding redundant reloads. We implement RelayGR on Huawei Ascend NPUs and evaluate it with real queries. Under a fixed P99 SLO, RelayGR supports up to 1.5× longer sequences and improves SLO-compliant throughput by up to 3.6×.