mlsanityData to 14 Oct 2026

SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning

arXiv 2506.01096 ↗First posted 1 Jun 2025cs.AIcited by 18

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajectories, leading to inefficient learning. At the same time, these offline trajectories that represent correct reasoning paths are not utilized by standard on-policy reinforcement learning methods. We introduce SuperRL, a unified training framework that adaptively alternates between RL and SFT. Whenever every rollout for a given instance receives zero reward, indicating the absence of a learning signal, SuperRL falls back to SFT on the curated offline data. Extensive experiments across diverse reasoning benchmarks show that SuperRL surpasses vanilla RL by delivering higher sample efficiency, stronger generalization, and improved robustness under sparse rewards.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 5 of its 34 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

5 of 34 references.

References · 5

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  4. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  5. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approachcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  2. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  3. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  4. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  5. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  6. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  7. 2026AgenticRed: Evolving Agentic Systems for Red-Teamingcited by 5
  8. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 5
  9. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  10. 2026SD-E2: Semantic Exploration for Reasoning Under Token Budgetscited by 3
  11. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoningcited by 1
  12. 2026BRIDGE: Bridging Reasoning In Distillation Gap Elimination via Structure-Aware Maskingcited by 0
6 more
  1. 2026Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verificationcited by 0
  2. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  3. 2026AIR: Adaptive Interleaved Reasoning with Code in MLLMscited by 0
  4. 2026MedRouter: Demystifying Knowledge Differences Across Medical LLMs for Routing-Based Reasoningcited by 0
  5. 2026Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RLcited by 0
  6. 2026PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
  2. 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
  3. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
  4. 2025TTRL: Test-Time Reinforcement Learning
  5. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
  6. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression