mlsanityData to 7 Oct 2026

Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

arXiv 2510.25992 ↗First posted 29 Oct 2025cs.CL cs.AI cs.LGcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) often struggle with problems that require multi-step reasoning. For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to overfit long demonstrations through rigid token-by-token imitation. To address this gap, we propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions". SRL trains the model to generate an internal reasoning monologue before committing to each action. It provides smoother rewards based on the similarity between the model's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As a result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning-oriented LLMs.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 50 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

39 of 50 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning5 Aug
  2. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  5. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  10. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 27

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2025s1: Simple test-time scalingcited by 1,311
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  7. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  8. 2025LIMO: Less is More for Reasoningcited by 488
  9. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  10. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  11. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  12. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
15 more
  1. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  2. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  3. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  4. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  5. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  6. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  7. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  8. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  9. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  10. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  11. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  12. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoningcited by 40
  13. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillationcited by 39
  14. 2025Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planningcited by 15
  15. 2025On Teacher Hacking in Language Model Distillationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Groundingcited by 9
  2. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Searchcited by 6
  3. 2025Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewardscited by 5
  4. 2026AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineeringcited by 4
  5. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  6. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  7. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branchescited by 1
  8. 2026From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoningcited by 0
  9. 2026AIPO: Learning to Reason from Active Interactioncited by 0
  10. 2026DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learningcited by 0
  11. 2026RMSWeb: Reflection, Failure-Mode Mining, and Salvage-DS for Web Agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  2. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
  3. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  4. 2026Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge
  5. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  6. 2026Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
  7. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
  8. 2026TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFT