mlsanityData to 14 Oct 2026

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

arXiv 2507.00018 ↗First posted 15 Jun 2025cs.LG cs.AI cs.CLcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to 25% relative gain and 6% absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 46 references that are in the map. First posted on arXiv 15 Jun 2025.

Builds on

33 of 46 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Learning Dynamics of LLM Finetuning15 Jul
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  5. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning18 Mar
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2016Generative Adversarial Imitation Learningcited by 1,597
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
12 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  4. 2025LIMO: Less is More for Reasoningcited by 498
  5. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  8. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  9. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  10. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  11. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13
  12. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  2. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  3. 2026Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgettingcited by 16
  4. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
  5. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidancecited by 7
  6. 2025RealDPO: Real or Not Real, that is the Preferencecited by 6
  7. 2025Sample-efficient LLM Optimization with Reset Replaycited by 4
  8. 2025Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimizationcited by 3
  9. 2026Knowledge Graph-Assisted LLM Post-Training for Enhanced Legal Reasoningcited by 3
  10. 2026Compatibility-Aware Dynamic Fine-Tuning for Large Language Modelscited by 3
  11. 2025SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compressioncited by 2
  12. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
5 more
  1. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  2. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  3. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  4. 2026RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflectioncited by 0
  5. 2026Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  2. 2025Explicit Preference Optimization: No Need for an Implicit Reward Model
  3. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  4. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
  5. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
  6. 2024Understanding Reference Policies in Direct Preference Optimization
  7. 2025Reinforcement learning fine-tuning of language model for instruction following and math reasoning
  8. 2026What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training