# Efficient Inference for Large Language Model-based Generative Recommendation

arXiv: https://arxiv.org/abs/2410.05165 · HTML: https://mlsanity.com/paper/2410.05165/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 43)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Recommender Systems with Generative Retrieval](https://mlsanity.com/paper/2305.05065/) (2023)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [Data-efficient Fine-tuning for LLM-based Recommendation](https://mlsanity.com/paper/2401.17197/) (2024)
- [How to Index Item IDs for Recommendation Foundation Models](https://mlsanity.com/paper/2305.06569/) (2023)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023)
- [Mining Latent Structures for Multimedia Recommendation](https://mlsanity.com/paper/2104.09036/) (2021)
- [How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?](https://mlsanity.com/paper/1511.05101/) (2015)
- [GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding](https://mlsanity.com/paper/2402.02082/) (2024)
- [A Bi-Step Grounding Paradigm for Large Language Models in Recommendation Systems](https://mlsanity.com/paper/2308.08434/) (2023)

## Built on by (most cited first, 11 of 11)

- [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025)
- [Collaboration of Large Language Models and Small Recommendation Models for Device-Cloud Recommendation](https://mlsanity.com/paper/2501.05647/) (2025)
- [OneSearch-V2: The Latent Reasoning Enhanced Self-distillation Generative Search Framework](https://mlsanity.com/paper/2603.24422/) (2026)
- [IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation](https://mlsanity.com/paper/2506.13229/) (2025)
- [DeepInterestGR: Mining Deep Multi-Interest Using Multi-Modal LLMs for Generative Recommendation](https://mlsanity.com/paper/2602.18907/) (2026)
- [xGR: Efficient Generative Recommendation Serving at Scale](https://mlsanity.com/paper/2512.11529/) (2025)
- [Test-Time Scaling Strategies for Generative Retrieval in Multimodal Conversational Recommendations](https://mlsanity.com/paper/2508.18132/) (2025)
- [Give Users the Wheel: Towards Promptable Recommendation Paradigm](https://mlsanity.com/paper/2602.18929/) (2026)
- [Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation](https://mlsanity.com/paper/2604.27747/) (2026)
- [On the Memorization Behavior of LLMs in Generative Recommendation: Observations, Implications, and Training Strategies](https://mlsanity.com/paper/2606.17276/) (2026)
- [FedCGR: Federated Cross-Domain Generative Recommendation](https://mlsanity.com/paper/2608.10929/) (2026)

## Abstract (arXiv)

Large Language Model (LLM)-based generative recommendation has achieved notable success, yet its practical deployment is costly particularly due to excessive inference latency caused by autoregressive decoding. For lossless LLM decoding acceleration, Speculative Decoding (SD) has emerged as a promising solution. However, applying SD to generative recommendation presents unique challenges due to the requirement of generating top-K items (i.e., K distinct token sequences) as a recommendation list by beam search. This leads to more stringent verification in SD, where all the top-K sequences from the target LLM must be successfully drafted by the draft model at each decoding step. To alleviate this, we consider 1) boosting top-K sequence alignment between the draft model and the target LLM, and 2) relaxing the verification strategy to reduce trivial LLM calls. To this end, we propose an alignment framework named AtSpeed, which presents the AtSpeed-S optimization objective for top-K alignment under the strict top-K verification. Moreover, we introduce a relaxed sampling verification strategy that allows high-probability non-top-K drafted sequences to be accepted, significantly reducing LLM calls. Correspondingly, we propose AtSpeed-R for top-K alignment under this relaxed sampling verification. Empirical results on two real-world datasets demonstrate that AtSpeed significantly accelerates LLM-based generative recommendation, e.g., near 2x speedup under strict top-K verification and up to 2.5x speedup under relaxed sampling verification. The codes and datasets are released at https://github.com/Linxyhaha/AtSpeed.