mlsanityData to 14 Oct 2026

VERPO: Verified Evidence Regularized Policy Optimization

arXiv 2609.06100 ↗First posted 5 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Verifiable rewards improve language models through reliable task-level feedback, but methods based on Group Relative Policy Optimization (GRPO) apply a sequence-level advantage uniformly across all tokens. This coarse credit assignment reinforces or penalizes entire responses without identifying which local decisions to preserve, reinforce, or revise. Conversely, evidence-conditioned self-distillation provides denser token-level supervision, yet teacher imitation can transfer stylistic artifacts and miscalibrated confidence that destabilize training when misaligned with task success. We introduce VERPO, which converts evidence-conditioned guidance into reward-aligned token-level credit assignment while retaining the outcome objective. VERPO decomposes teacher guidance into an evidence-free reference term and signed, evidence-induced corrections at each token. A stopped controller combines selective acceptance, token-wise localization, and cost-aware scaling by balancing alignment with the local GRPO update direction against Fisher movement cost. Furthermore, we introduce Fisher Evidence Contrast (FEC), which attenuates nuisance shifts along an estimated evidence-presence direction through a regularized projection. Across five scientific reasoning and tool-use tasks, VERPO prevents optimization collapse and consistently achieves the highest multi-task average across model backbones, yielding marked improvements particularly on smaller models over strong baselines. Qualitative diagnostics confirm that token acceptance selectively targets reasoning bottlenecks consistent with local reward alignment and Fisher movement cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Sep 2026.

Builds on

27 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization19 May
  2. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing2 Apr
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  9. 2026Reinforcement Learning via Self-Distillationcited by 332
  10. 2026Self-Distilled RLVRcited by 131
  11. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  12. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 45
8 more
  1. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 39
  2. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 25
  3. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 18
  4. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  5. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillationcited by 7
  6. 2026EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillationcited by 6
  7. 2026When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillationcited by 5
  8. 2026EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
  2. 2026Policy-Conditioned Counterfactual Credit for Verifiable Reinforcement Learning of Long-Horizon Language Agents
  3. 2026Reward-Aligned Distillation
  4. 2026Function-Space Temporal Self-Distillation for Policy Optimization
  5. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  6. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  7. 2026SocraticPO: Policy Optimization via Interactive Guidance
  8. 2026Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs