# F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking

arXiv: https://arxiv.org/abs/2605.12995 · HTML: https://mlsanity.com/paper/2605.12995/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 76 references that are in the map.
- First posted on arXiv 13 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning](https://mlsanity.com/paper/2503.06034/) (2025)
- [Zero-Shot Next-Item Recommendation using Large Pretrained Language Models](https://mlsanity.com/paper/2304.03153/) (2023)
- [LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking](https://mlsanity.com/paper/2311.02089/) (2023)
- [Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2602.02710/) (2026)
- [GVPO: Group Variance Policy Optimization for Large Language Model Post-Training](https://mlsanity.com/paper/2504.19599/) (2025)
- [Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models](https://mlsanity.com/paper/2312.16098/) (2023)
- [Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2503.16734/) (2025)
- [Personalized Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2412.02142/) (2024)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations](https://mlsanity.com/paper/2505.14106/) (2025)
- [Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamics](https://mlsanity.com/paper/2509.00190/) (2025)
- [WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning](https://mlsanity.com/paper/2602.17025/) (2026)
- [Pluralistic Off-policy Evaluation and Alignment](https://mlsanity.com/paper/2509.19333/) (2025)
- [Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning](https://mlsanity.com/paper/2510.20150/) (2025)
- [SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes](https://mlsanity.com/paper/2601.05600/) (2026)
- [Active Learning for Direct Preference Optimization](https://mlsanity.com/paper/2503.01076/) (2025)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [Policy-Gradient Training of Language Models for Ranking](https://mlsanity.com/paper/2310.04407/) (2023)
- [An Empirical Study on Zero-Data Bootstrapping for Conversational Recommender Systems](https://mlsanity.com/paper/2504.15476/) (2025)
- [Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization](https://mlsanity.com/paper/2509.25717/) (2025)
- [Evaluation on Entity Matching in Recommender Systems](https://mlsanity.com/paper/2601.17218/) (2026)
- [HiGR: Industrial-Scale Hierarchical Generative Slate Recommendation Framework in Tencent](https://mlsanity.com/paper/2512.24787/) (2025)

## Built on by (most cited first, 1 of 1)

- [RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts](https://mlsanity.com/paper/2607.18470/) (2026)

## Abstract (arXiv)

Traditional retrieval pipelines optimize utility through stages of candidate retrieval and reranking, where ranking operates over a predefined candidate set. Large Language Models (LLMs) broaden this into a generative process: given a candidate pool, an LLM can generate a subset and order it within a single autoregressive pass. However, this flexibility introduces a new optimization challenge: the model must search a combinatorial output space while receiving utility feedback only after the full ranked list is generated. Because this feedback is defined over the completed sequence, it cannot distinguish whether a poor result arises from failing to generate a relevant subset or from failing to rank that subset correctly. This credit assignment gap makes end-to-end optimization unstable and sample-inefficient. Existing systems often address this by separating candidate generation from ranking. However, such decoupling remains misaligned with downstream utility because ranking is limited by the candidate set it receives. To bridge this gap, we propose a unified framework that performs both within a single autoregressive rollout and optimizes them end-to-end via factorized group-relative policy optimization (F-GRPO). Our framework factorizes the policy into candidate generation and ranking while sharing a single LLM backbone, and jointly trains them with an order-invariant coverage reward and a position-aware utility reward. To address the resulting phase-specific credit assignment problem, we use separate group-relative advantages for generation and ranking within a two-phase sequence-level objective. Across sequential recommendation and multi-hop question answering benchmarks, F-GRPO improves top-ranked performance over GRPO and decoupled baselines, outperforms supervised alternatives, and remains competitive with strong zero-shot rerankers, with no architectural changes at inference time.