# Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendation

arXiv: https://arxiv.org/abs/2602.10699 · HTML: https://mlsanity.com/paper/2602.10699/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Recommender Systems with Generative Retrieval](https://mlsanity.com/paper/2305.05065/) (2023)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [A Bi-Step Grounding Paradigm for Large Language Models in Recommendation Systems](https://mlsanity.com/paper/2308.08434/) (2023)
- [On Softmax Direct Preference Optimization for Recommendation](https://mlsanity.com/paper/2406.09215/) (2024)
- [EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaboration](https://mlsanity.com/paper/2406.14017/) (2024)
- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [SDPO: Segment-Level Direct Preference Optimization for Social Agents](https://mlsanity.com/paper/2501.01821/) (2025)
- [Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology](https://mlsanity.com/paper/1905.12767/) (2019)
- [REG4Rec: Reasoning-Enhanced Generative Model for Large-Scale Recommendation Systems](https://mlsanity.com/paper/2508.15308/) (2025)
- [MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation](https://mlsanity.com/paper/2510.24431/) (2025)
- [Decoding Matters: Addressing Amplification Bias and Homogeneity Issue for LLM-based Recommendation](https://mlsanity.com/paper/2406.14900/) (2024)

## Built on by (most cited first, 2 of 2)

- [Difficulty-Aware Semantic-ID Optimization for Generative Recommendation](https://mlsanity.com/paper/2608.20611/) (2026)
- [ReCast: Recasting Learning Signals for Reinforcement Learning in Generative Recommendation](https://mlsanity.com/paper/2604.22169/) (2026)

## Abstract (arXiv)

Generative recommendation via autoregressive models has unified retrieval and ranking into a single conditional generation framework. However, fine-tuning these models with Reinforcement Learning (RL) often suffers from a fundamental probability-reward mismatch. Conventional likelihood-dominated decoding (e.g., beam search) exhibits a myopic bias toward locally probable prefixes, which causes two critical failures: (1) insufficient exploration, where high-reward items in low-probability branches are prematurely pruned and rarely sampled, and (2) advantage compression, where trajectories sharing high-probability prefixes receive highly correlated rewards with low within-group variance, yielding a weak comparative signal for RL. To address these challenges, we propose V-STAR, a Value-guided Sampling and Tree-structured Advantage Reinforcement framework. V-STAR forms a self-evolving loop via two synergistic components. First, a Value-Guided Efficient Decoding (VED) is developed to identify decisive nodes and selectively deepen high-potential prefixes. This improves exploration efficiency without exhaustive tree search. Second, we propose Sibling-GRPO, which exploits the induced tree topology to compute sibling-relative advantages and concentrates learning signals on decisive branching decisions. Extensive experiments on both offline and online datasets demonstrate that V-STAR outperforms state-of-the-art baselines, delivering superior accuracy and candidate-set diversity under strict latency constraints.