mlsanityData to 7 Oct 2026

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

arXiv 2604.08926 ↗First posted 10 Apr 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose DYPO (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a textitGroup Alignment Loss (GAL) that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a textitMulti-Teacher Distillation mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a textitDynamic Exploitation-Exploration Gating mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8% on complex reasoning benchmarks and 13.3% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 39 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

27 of 39 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning8 Sep
  2. 2025A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning11 Jul
  3. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  4. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  5. 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning1 Jun
  6. 2025Learning to Reason under Off-Policy Guidance21 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025Process Reinforcement through Implicit Rewards3 Feb
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2024OpenAI o1 System Cardcited by 1,220
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  12. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
3 more
  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  2. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  3. 2023Aligning Large Language Models with Human: A Surveycited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Trust-Region Adaptive Policy Optimization
  2. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
  3. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning
  4. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  5. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning
  6. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  7. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  8. 2025Single-stream Policy Optimization