# SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation

arXiv: https://arxiv.org/abs/2605.17648 · HTML: https://mlsanity.com/paper/2605.17648/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 27 of 27)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recommender Systems with Generative Retrieval](https://mlsanity.com/paper/2305.05065/) (2023)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Learnable Item Tokenization for Generative Recommendation](https://mlsanity.com/paper/2405.07314/) (2024)
- [Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation](https://mlsanity.com/paper/2503.22675/) (2025)
- [OneRec-Think: In-Text Reasoning for Generative Recommendation](https://mlsanity.com/paper/2510.11639/) (2025)
- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [R2ec: Towards Large Recommender Models with Reasoning](https://mlsanity.com/paper/2505.16994/) (2025)
- [Reasoning over Semantic IDs Enhances Generative Recommendation](https://mlsanity.com/paper/2603.23183/) (2026)
- [Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning](https://mlsanity.com/paper/2510.20150/) (2025)
- [Why Thinking Hurts: Diagnosing and Rectifying Linguistic Inertia in Large Language Models for Recommendation](https://mlsanity.com/paper/2602.16587/) (2026)
- [RecCoT: Enhancing Recommendation via Chain-of-Thought](https://mlsanity.com/paper/2506.21032/) (2025)

## Built on by (most cited first, 2 of 2)

- [AnyJev Technical Report](https://mlsanity.com/paper/2610.00831/) (2026)
- [The Disconnect Between Better Descriptive Reasoning Trace Quality and Recommendation Effectiveness](https://mlsanity.com/paper/2608.23154/) (2026)

## Abstract (arXiv)

Generative recommendation treats next-item prediction as autoregressive item-identifier generation. Specifically, items are encoded as semantic identifiers (SIDs), which are short coarse-to-fine token sequences whose early tokens capture broad semantics and later tokens refine them. Recent work augments this paradigm with reasoning traces and optimizes them via reinforcement learning with verifiable rewards, typically outcome-reward algorithm with exact-match feedback on the generated SID. However, in large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct; when a generated SID mismatches, outcome-reward cannot identify which SID-token prediction caused the mismatch and may penalize matched SID-token positions together with the mismatched position. We identify that the natural unit of credit assignment in this setting is a single reasoning step (one thinking block paired with one SID token). We instantiate this idea in SAPO (Step-Aligned Policy Optimization): rather than broadcasting one advantage to the whole response, SAPO computes a separate group-relative advantage for each reasoning step and applies it only to the corresponding thinking block and SID token. Across three real-world recommendation datasets, SAPO stabilizes reinforcement-learning training and consistently improves over existing generative recommendation baselines, with the largest gains where sparse exact-match feedback makes reasoning-step credit assignment important. Our results suggest that reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output.