mlsanityData to 14 Oct 2026

Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL

arXiv 2610.11128 ↗First posted 8 Oct 2026cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking also fits: Reinforcement learning for reasoning

Abstract (arXiv)

A verified reference solution provides a correct trajectory for training a reasoning model. Alternatively, a prefix of the reference can guide the model in generating a trajectory of its own. How much reference guidance should we provide? We study this question through prefix continuation, where the model continues from a reference prefix and keeps the resulting trajectory if it passes verification, falling back to the reference otherwise. Since both procedures produce correct trajectories, we compare their distributions with the ideal distribution, the model's own distribution conditioned on successful verification. For one continuation, we derive the KL divergence in closed form, which, up to a bounded term, decreases with the product of the probability of generating a different correct trajectory and the reference surprisal, the negative log probability of the reference suffix given the prefix. Since a longer prefix tends to raise the former but lowers the latter, continuation success alone does not determine the preferred amount of guidance. From this analysis, we learn a prefix selector shared across training questions from continuation outcomes, without estimating success probabilities or additional generation. The resulting Adaptive Reference Guidance (ARG) constructs correct trajectories within a fixed generation budget, and we apply it to all-failure groups in Group Relative Policy Optimization (GRPO). Experiments on Qwen3-4B and Qwen3-8B across five mathematical reasoning benchmarks show that ARG achieves the highest aggregate pass@12 among the evaluated methods with competitive average sampled accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 34 of its 61 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

34 of 61 references.

References · 34

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  9. 2026Reinforcement Learning via Self-Distillationcited by 332
  10. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  11. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  12. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 65
22 more
  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  2. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  3. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  4. 2026Maximum Likelihood Reinforcement Learningcited by 36
  5. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learningcited by 34
  6. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 32
  7. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 20
  8. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  9. 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoningcited by 18
  10. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Modelscited by 18
  11. 2025What is the objective of reasoning with reinforcement learning?cited by 18
  12. 2025Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffoldingcited by 16
  13. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  14. 2026Learning to Hint for Reinforcement Learningcited by 12
  15. 2026Learning Robust Reasoning through Guided Adversarial Self-Playcited by 8
  16. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Workscited by 6
  17. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  18. 2026Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problemscited by 3
  19. 2026Tail-Likelihood Reinforcement Learningcited by 3
  20. 2026Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoningcited by 2
  21. 2026From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoningcited by 2
  22. 2026Binary Rewards and Reinforcement Learning: Fundamental Challengescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Directional Reasoning Trajectory Change (DRTC): Identifying Critical Trace Segments in Reasoning Models
  2. 2026Reasoning Models Don't Just Think Longer, They Move Differently
  3. 2026Learn Hard Problems During RL with Reference Guided Fine-tuning
  4. 2026TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stopping
  5. 2026Revisiting Complete Reasoning Traces for Post-Training
  6. 2026Beyond Rejection Sampling: Trajectory Fusion for Scaling Mathematical Reasoning
  7. 2026Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
  8. 2026REWIND: Uncertainty-Guided Trajectory Repair for Test-Time Reasoning