Data to 14 Oct 2026

Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

arXiv 2507.12856 ↗First posted 17 Jul 2025cs.LG cs.AIcited by 38

On the map: Offline RL and imitation learning from data → Imitation learning methods

also in 1 more

Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 59 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

38 of 59 references.

References · 38

2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
2024Qwen2.5 Technical Reportcited by 4,942
2015Continuous control with deep reinforcement learningcited by 3,407
2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
2019Fine-Tuning Language Models from Human Preferencescited by 2,150

26 more

2017Decoupled Weight Decay Regularizationcited by 1,831
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2025s1: Simple test-time scalingcited by 1,341
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
2018Maximum a Posteriori Policy Optimisationcited by 328
2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 81
2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
2018Policy Optimization via Importance Samplingcited by 47
2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
2016Tighter bounds lead to improved classifierscited by 3
2016Efficient iterative policy optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 25.

38 papers.

Sort:

By area:

2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 19
2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
2025Proximal Supervised Fine-Tuningcited by 16
2025Anchored Supervised Fine-Tuningcited by 15
2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capabilitycited by 14
2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
2025Debunk the Myth of SFT Generalizationcited by 8
2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8

26 more

2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
2025Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuningcited by 3
2026RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuningcited by 3
2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
2026Rotation-Preserving Supervised Fine-Tuningcited by 3
2026Compatibility-Aware Dynamic Fine-Tuning for Large Language Modelscited by 3
2026A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Trainingcited by 3
2026PriFT: Prior-Support Guided Supervised Fine-Tuningcited by 2
2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
2025SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAGcited by 1
2026GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectificationcited by 1
2026Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectorscited by 1
2026Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descentcited by 1
2026Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Modelscited by 1
2026DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimizationcited by 1
2026Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoningcited by 1
2025VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervisioncited by 0
2026Goal-Conditioned Supervised Learning for LLM Fine-Tuningcited by 0
2026A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Designcited by 0
2026Online Self-Weighted Fine-Tuningcited by 0
2026On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Trainingcited by 0
2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policynesscited by 0
2026Small Agents with Semantic Search: Efficient Multilingual Code Localizationcited by 0

Similar, not cited

Close in wording, no citation either way