mlsanityData to 14 Oct 2026

Selective Off-Policy Reference Tuning with Plan Guidance

arXiv 2605.11505 ↗First posted 12 May 2026cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards helps reasoning, but GRPO-style methods stall on hard prompts where all sampled rollouts fail. SORT adds a repair update for those failures without changing rollout generation: it derives a plan from the reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This turns all-wrong prompts into selective, structure-aware learning signals instead of uniform imitation. Across three backbones and eight reasoning benchmarks, SORT improves over GRPO and guidance baselines, with largest gains on weaker models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 40 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

31 of 40 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  2. 2025Nudging the Boundaries of LLM Reasoning30 Sep
  3. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  4. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  7. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO16 May
  8. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  9. 2025Learning to Reason under Off-Policy Guidance21 Apr
  10. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  11. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning10 Feb
  12. 2025Process Reinforcement through Implicit Rewards3 Feb
2 more
  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  2. 2024Free Process Rewards without Process Labels2 Dec

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  10. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  11. 2026Reinforcement Learning via Self-Distillationcited by 332
  12. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
5 more
  1. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  2. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
  3. 2025Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocationcited by 22
  4. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  5. 2026SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoningcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
  2. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts
  3. 2026Learning to Hint for Reinforcement Learning
  4. 2026From Outcomes to Strategies: Learning Strategy Utility for Mathematical Reasoning
  5. 2026GRPODropout: Less is More for Online Reinforcement Learning Rollouts
  6. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewards
  7. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
  8. 2026Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization