# Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning

arXiv: https://arxiv.org/abs/2510.20150 · HTML: https://mlsanity.com/paper/2510.20150/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in LLM reasoning for recommendation, one of 17 directions in LLM reasoning.
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Placed in

- LLM reasoning / [LLM reasoning for recommendation](https://mlsanity.com/lr/llm-reasoning-for-recommendation/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [RecLLM-R1: A Two-Stage Training Paradigm with Reinforcement Learning and Chain-of-Thought v1](https://mlsanity.com/paper/2506.19235/) (2025)

## Built on by (most cited first, 11 of 11)

- [SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation](https://mlsanity.com/paper/2605.17648/) (2026)
- [Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2601.13115/) (2026)
- [Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping](https://mlsanity.com/paper/2603.06923/) (2026)
- [F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking](https://mlsanity.com/paper/2605.12995/) (2026)
- [Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation](https://mlsanity.com/paper/2608.11980/) (2026)
- [Difficulty-Aware Semantic-ID Optimization for Generative Recommendation](https://mlsanity.com/paper/2608.20611/) (2026)
- [FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning](https://mlsanity.com/paper/2603.11901/) (2026)
- [Enhancing Local Life Service Recommendation with Agentic Reasoning in Large Language Model](https://mlsanity.com/paper/2604.14051/) (2026)
- [RRCM: Ranking-Driven Retrieval over Collaborative and Meta Memories for LLM Recommendation](https://mlsanity.com/paper/2605.07129/) (2026)
- [Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation](https://mlsanity.com/paper/2606.08480/) (2026)
- [Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders](https://mlsanity.com/paper/2610.17771/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs to the recommendation task remains challenging: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, a two-stage framework for end-to-end training of LLM-based conversational recommender systems. In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded demonstrations from powerful blackbox LLMs to warm-start the RL training. In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead of token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and introducing a rank-level importance ratio based on the geometric mean of rank-wise token probabilities to stabilize policy updates. Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released at https://github.com/yaochenzhu/Rank-GRPO.