mlsanityData to 7 Oct 2026

RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents

arXiv 2602.03025 ↗First posted 3 Feb 2026cs.AI cs.CLcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Multi-turn tool calling is challenging for Large Language Models (LLMs) because rewards are sparse and exploration is expensive. A common recipe, SFT followed by GRPO, can stall when within-group reward variation is low (e.g., more rollouts in a group receive the all 0 or all 1 reward), making the group-normalized advantage uninformative and yielding vanishing updates. To address this problem, we propose RC-GRPO (Reward-Conditioned Group Relative Policy Optimization), which treats exploration as a controllable steering problem via discrete reward tokens. We first fine-tune a Reward-Conditioned Trajectory Policy (RCTP) on mixed-quality trajectories with reward goal special tokens (e.g., <|high_reward|>, <|low_reward|>) injected into the prompts, enabling the model to learn how to generate distinct quality trajectories on demand. Then during RL, we sample diverse reward tokens within each GRPO group and condition rollouts on the sampled token to improve within-group diversity, improving advantage gains. On the Berkeley Function Calling Leaderboard v4 (BFCLv4) multi-turn benchmark, our method yields consistently improved performance than baselines, and the performance on Qwen-2.5-7B-Instruct even surpasses all closed-source API models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 28 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

22 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
7 more
  1. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  2. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  3. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  4. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  5. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  6. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  7. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 12
  2. 2026Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learningcited by 4
  3. 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 4
  4. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Workscited by 3
  5. 2026SCORP: Scene-Consistent Multi-agent Diffusion Planning with Stable Online Reinforcement Post-Training for Cooperative Drivingcited by 2
  6. 2026Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learningcited by 2
  7. 2026Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Callingcited by 1
  8. 2026Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolutioncited by 0
  9. 2026Can the Environment Speak for Itself? T2-GRPO: A Turn-Trajectory Group Relative Policy Optimization for Caregiver Agentscited by 0
  10. 2026SCOUT: Self-Checking and Recovery-Aware Tool-Thought Agents for Ultra-Long Egocentric Video Reasoningcited by 0
  11. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  3. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  4. 2026RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
  5. 2026Reinforcement Learning with Decomposed Subtasks
  6. 2026TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization
  7. 2026GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
  8. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization