mlsanityData to 13 Oct 2026

Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

arXiv 2609.36864 ↗First posted 29 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5× reduction in generated tokens and a 1.8× speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

26 of 33 references.

References · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2025Qwen3 Technical Reportcited by 8,510
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,729
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,369
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,304
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,201
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,111
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,020
  10. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 548
  11. 2026Reinforcement Learning via Self-Distillationcited by 310
  12. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 309
14 more
  1. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 268
  2. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 240
  3. 2026Self-Distilled RLVRcited by 121
  4. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 76
  5. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 66
  6. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 49
  7. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  8. 2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Costcited by 12
  9. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agentscited by 9
  10. 2026Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learningcited by 9
  11. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoningcited by 5
  12. 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 5
  13. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 3
  14. 2026Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
  2. 2026Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
  3. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
  4. 2026Reinforcement Learning with Decomposed Subtasks
  5. 2026Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance
  6. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training
  7. 2026Learning More from Less: Reinforcement Learning from Hindsight
  8. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPO