mlsanityData to 13 Oct 2026

Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating

arXiv 2603.11321 ↗First posted 11 Mar 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards improves reasoning in large language models, yet on-policy learning often suffers from cold-start challenges in sparse-reward settings. Recent mixed-policy approaches address this by combining off-policy teacher data with on-policy training. However, simply combining these introduce a persistent off-policy gradient mass that risks training collapse and instability. To address this challenge, we propose Hindsight-Anchored Policy Optimization (HAPO), a framework that allows teacher intervention to act as a temporary support. HAPO employs Beta-Binomial confidence gating, an adaptive gating mechanism that decides when to open the gate for teacher intervention. The intervention operates with Synthetic Success Injection, which replaces the group's lowest-reward rollout with a verified teacher trajectory. We also introduce adaptive threshold annealing, which gradually retracts the support and restores on-policy training within a finite horizon to mitigate persistent off-policy drift. We demonstrate that HAPO can be layered on top of existing mixed-policy methods in a generalizable manner. Across six math reasoning benchmarks and two model scales, HAPO improves the average accuracy of three major mixed-policy methods while maintaining training stability.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 27 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

21 of 27 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  2. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  3. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  4. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  5. 2025Learning to Reason under Off-Policy Guidance21 Apr
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,003
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,369
  8. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 167
  9. 2017Hindsight policy gradientscited by 37
  10. 2019Hindsight Trust Region Policy Optimizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  2. 2026Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
  3. 2026SocraticPO: Policy Optimization via Interactive Guidance
  4. 2026TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFT
  5. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training
  6. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  7. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  8. 2026Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning