mlsanityData to 14 Oct 2026

On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting

arXiv 2508.11408 ↗First posted 15 Aug 2025cs.LG cs.AIcited by 91

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs). Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data. To address this, we present a novel investigation into the unified view of SFT and RL through an off-policy versus on-policy lens. We propose CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting, which reframes SFT not as a separate stage but as a dynamically weighted auxiliary objective within the on-policy RL process. Based on an analysis of off-policy expert data's influence at both holistic and granular levels, we incorporate a dual-control mechanism in CHORD. Specifically, the framework first employs a global coefficient to holistically guide the transition from off-policy imitation to on-policy exploration, and then applies a token-wise weighting function that enables granular learning from the expert, which promotes on-policy exploration and mitigates disruption from off-policy data. We conduct extensive experiments across various practical tasks, providing empirical evidence that CHORD achieves a stable and efficient learning process. By effectively harmonizing off-policy expert data with on-policy exploration, CHORD demonstrates significant improvements over baselines. We release the implementation at https://github.com/modelscope/Trinity-RFT/tree/main/examples/mix_chord to inspire further research.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 59 of its 69 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

59 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification7 Aug
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains23 Jul
  3. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)17 Jul
  4. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards25 Jun
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  6. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training28 Jan

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  12. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
38 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  3. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  12. 2024Yi: Open Foundation Models by 01.AIcited by 719
  13. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  14. 2025LIMO: Less is More for Reasoningcited by 498
  15. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  16. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  17. 2025Learning to Reason under Off-Policy Guidancecited by 281
  18. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  19. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  20. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  21. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  22. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  23. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  24. 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 121
  25. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  26. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  27. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  28. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 63
  29. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  30. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  31. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  32. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  33. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  34. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  35. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  36. 2025Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoningcited by 15
  37. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11
  38. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 61.

91 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  4. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  5. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  6. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 35
  7. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  8. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  9. 2025Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Samplingcited by 18
  10. 2025Proximal Supervised Fine-Tuningcited by 16
  11. 2025Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffoldingcited by 16
  12. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
38 more
  1. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  2. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  3. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 14
  4. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergencecited by 13
  5. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Samplingcited by 13
  6. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  7. 2025Scaling Generalist Data-Analytic Agentscited by 11
  8. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
  9. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  10. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 11
  11. 2025DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimizationcited by 9
  12. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  13. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  14. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMscited by 7
  15. 2026SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuningcited by 7
  16. 2026Generative Recommendation for Large-Scale Advertisingcited by 7
  17. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  18. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  19. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimizationcited by 6
  20. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  21. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
  22. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 5
  23. 2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevancecited by 4
  24. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  25. 2026Visual Reasoning through Tool-supervised Reinforcement Learningcited by 4
  26. 2026Learning Agentic Policy from Action Guidancecited by 4
  27. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  28. 2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
  29. 2025CALM Before the STORM: Unlocking Native Reasoning for Optimization Modelingcited by 3
  30. 2025Do Not Step Into the Same River Twice: Learning to Reason from Trial and Errorcited by 3
  31. 2025Toward Honest Language Models for Deductive Reasoningcited by 3
  32. 2025Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimizationcited by 3
  33. 2026Entropy-Gated Selective Policy Optimization:Token-Level Gradient Allocation for Hybrid Training of Large Language Modelscited by 3
  34. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  35. 2026WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Trainingcited by 3
  36. 2026PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agentscited by 3
  37. 2025RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initializationcited by 2
  38. 2025Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generationcited by 2

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
  2. 2026TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFT
  3. 2025Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning
  4. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  5. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
  6. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
  7. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  8. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training