mlsanityData to 7 Oct 2026

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

arXiv 2606.25852 ↗First posted 24 Jun 2026cs.LG cs.AIcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 43 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

31 of 43 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifier17 Apr
  2. 2026RTMC: Step-Level Credit Assignment via Rollout Trees13 Apr
  3. 2026Hindsight Credit Assignment for Long-Horizon LLM Agents7 Mar
  4. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  5. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  6. 2025Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs30 Sep
  7. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning26 Sep
  8. 2025Agentic Reinforcement Learning with Implicit Step Rewards23 Sep
  9. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents11 Sep
  10. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution27 May
  11. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  12. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
3 more
  1. 2025AgentRM: Enhancing Agent Generalization with Reward Modeling25 Feb
  2. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  3. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agents3 Jan

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
4 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2018Self-Imitation Learningcited by 135
  3. 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 56
  4. 2025Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPOcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Does Muon Help Agentic Reinforcement Learning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  2. 2025COPO: Consistency-Aware Policy Optimization
  3. 2026Learning from Viable Failure Prefixes: Milestone Viability Potential Policy Optimization for Long-Horizon LLM Agents
  4. 2026When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
  5. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
  6. 2026PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
  7. 2026Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
  8. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning