mlsanityData to 7 Oct 2026

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

arXiv 2404.14367 ↗First posted 22 Apr 2024cs.LGcited by 184

Not on the map. Why?

Cited by 116 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning. Different methods come with different implementation tradeoffs and performance differences, and existing empirical findings present different conclusions, for instance, some results show that online RL is quite important to attain good fine-tuning results, while others find (offline) contrastive or even purely supervised methods sufficient. This raises a natural question: what kind of approaches are important for fine-tuning with preference data and why? In this paper, we answer this question by performing a rigorous analysis of a number of fine-tuning techniques on didactic and full-scale LLM problems. Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i.e., employ a "negative gradient") outperform offline and maximum likelihood objectives. We conceptualize our insights and unify methods that use on-policy sampling or negative gradient under a notion of mode-seeking objectives for categorical distributions. Mode-seeking objectives are able to alter probability mass on specific bins of a categorical distribution at a fast rate compared to maximum likelihood, allowing them to relocate masses across bins more effectively. Our analysis prescribes actionable insights for preference fine-tuning of LLMs and informs how data should be collected for maximal improvement.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

43 of 61 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2024Self-Rewarding Language Modelscited by 654
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  12. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
31 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  10. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  12. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  13. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  14. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  15. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  16. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  17. 2023Nash Learning from Human Feedbackcited by 220
  18. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  19. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  20. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  21. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  22. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  23. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  24. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  25. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  26. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  27. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  28. 2023The Role of Baselines in Policy Gradient Optimizationcited by 25
  29. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 20
  30. 2023Aligning Language Models with Offline Learning from Human Feedbackcited by 14
  31. 2023SuperHF: Supervised Iterative Learning from Human Feedbackcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 69; 2025: 82; 2026: 33.

184 papers.

Sort:

By area:

  1. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  3. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  4. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  5. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  6. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  7. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  8. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  9. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  10. 2024Learning Dynamics of LLM Finetuningcited by 93
  11. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  12. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
38 more
  1. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  2. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  3. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  4. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  5. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  6. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  7. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  8. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  9. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  10. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  11. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  12. 2025Can Large Reasoning Models Self-Train?cited by 66
  13. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  14. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  15. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  16. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  17. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  18. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  19. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  20. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  21. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  22. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  23. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  24. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  25. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  26. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  27. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  28. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  29. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  30. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  31. 2025A Survey of Direct Preference Optimizationcited by 35
  32. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  33. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  34. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  35. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  36. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  37. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  38. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31

50 most cited of 184 All 184 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
  4. 2024Reinforcement Learning for LLM Post-Training: A Survey
  5. 2025Intelligently Weighting Multiple Reference Models for Direct Preference Optimization of LLMs
  6. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective
  7. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  8. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model