mlsanityData to 7 Oct 2026

All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning

arXiv 2503.01067 ↗First posted 3 Mar 2025cs.LGcited by 58

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via offline maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only lose information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a generation-verification gap, (1) it is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 63 of its 96 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

63 of 96 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  13. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  14. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  15. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  16. 2017Implicit Regularization in Matrix Factorizationcited by 318
  17. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  18. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  19. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  20. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  21. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  22. 2023Nash Learning from Human Feedbackcited by 220
  23. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  24. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  25. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  26. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  27. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  28. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  29. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  30. 2023Explaining grokking through circuit efficiencycited by 105
  31. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  32. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  33. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  34. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  35. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  36. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  37. 2018An elementary introduction to information geometrycited by 76
  38. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 18.

58 papers.

Sort:

By area:

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  2. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  5. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  6. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 64
  7. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  8. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  9. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  10. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  11. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  12. 2025Visual Test-time Scaling for GUI Agent Groundingcited by 27
38 more
  1. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  2. 2025Thought Manipulation: External Thought Can Be Efficient for Large Reasoning Modelscited by 23
  3. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  4. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  5. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  7. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  8. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  9. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  10. 2025DeepTrans: Deep Reasoning Translation via Reinforcement Learningcited by 8
  11. 2025Preference Learning with Lie Detectors can Induce Honesty or Evasioncited by 8
  12. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  13. 2025ProxyThinker: Test-Time Guidance through Small Visual Reasonerscited by 8
  14. 2025SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgementcited by 7
  15. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  16. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  17. 2026Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoningcited by 6
  18. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  19. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  20. 2025A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understandingcited by 5
  21. 2025RAST: Reasoning Activation in LLMs via Small-model Transfercited by 5
  22. 2025Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoningcited by 3
  23. 2025Why DPO is a Misspecified Estimator and How to Fix Itcited by 3
  24. 2026Compatibility-Aware Dynamic Fine-Tuning for Large Language Modelscited by 3
  25. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 3
  26. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2
  27. 2025Steering Information Utility in Key-Value Memory for Language Model Post-Trainingcited by 2
  28. 2025Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuningcited by 2
  29. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  30. 2025Generalization of RLVR Using Causal Reasoning as a Testbedcited by 2
  31. 2026Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Modelscited by 1
  32. 2026GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectificationcited by 1
  33. 2026The Future of Facts: Tracing the Factual Generation-Verification Gapcited by 1
  34. 2025Learning Parametric Distributions from Samples and Preferencescited by 0
  35. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  36. 2025From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimationcited by 0
  37. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  38. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  2. 2024It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF
  3. 2026Maximum Likelihood Reinforcement Learning
  4. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  5. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  8. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training