mlsanityData to 14 Oct 2026

SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

arXiv 2604.23747 ↗First posted 26 Apr 2026cs.LG cs.AI cs.CLcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 47 references that are in the map. First posted on arXiv 26 Apr 2026.

Builds on

38 of 47 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners6 Oct
  2. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs25 Sep
  3. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration21 Aug
  4. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  5. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  6. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  7. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  8. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  9. 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning1 Jun
  10. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  11. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning21 May
  12. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs19 May
8 more
  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
6 more
  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  3. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  4. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  5. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 58
  6. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 11
  2. 2026ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoningcited by 6
  3. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  4. 2026Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Trainingcited by 2
  5. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM trainingcited by 1
  6. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  7. 2026Understanding Human-like Solutions in Combinatorial Optimization via Learning and Searchcited by 0
  8. 2026Task Specialization Fine-Tuning for Contextual Reinforcement Learningcited by 0
  9. 2026Learning What to Imitate: Entropy-Aware Distribution Mixingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  2. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
  3. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  4. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
  5. 2026An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
  6. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning