mlsanityData to 14 Oct 2026

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

arXiv 2406.01462 ↗First posted 3 Jun 2024cs.LG cs.AI cs.CLcited by 55

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques -- online reinforcement learning (RL) such as Proximal Policy Optimization (PPO) and offline contrastive methods such as Direct Preference Optimization (DPO) -- were positioned as equivalent in prior work due to the fact that both have to start from the same offline preference dataset. To further expand our theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of dataset coverage, a concept that captures how the training data covers the test distribution and is widely used in RL. We prove that a global coverage condition is both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive a hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online data for KL regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than its pure offline counterpart DPO, while still preserving its computation and memory efficiency.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 67 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

51 of 67 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Robust Preference Optimization through Reward Model Distillation29 May
  2. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2020Learning to summarize from human feedbackcited by 2,588
29 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  10. 2023Nash Learning from Human Feedbackcited by 224
  11. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  12. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  13. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  14. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  15. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  16. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  17. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  18. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  19. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  20. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  21. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  22. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  23. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
  24. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  25. 2023Learning to Generate Better Than Your LLMcited by 49
  26. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  27. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  28. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  29. 2024Harnessing Density Ratios for Online Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 28; 2026: 22.

55 papers.

Sort:

By area:

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  2. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  3. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  4. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  5. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  6. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  7. 2025A Survey of Direct Preference Optimizationcited by 36
  8. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  9. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  10. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  11. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  12. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
38 more
  1. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 16
  2. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  3. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  4. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 11
  5. 2026Reward-free Alignment for Conflicting Objectivescited by 11
  6. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10
  7. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  8. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimizationcited by 9
  9. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  10. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  11. 2025Alignment of large language models with constrained learningcited by 8
  12. 2026Toward Native Multimodal Modeling: A Roadmapcited by 7
  13. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6
  14. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  15. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedbackcited by 6
  16. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
  17. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 6
  18. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 5
  19. 2025DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulationcited by 5
  20. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  21. 2025KL-regularization Itself is Differentially Private in Bandits and RLHFcited by 3
  22. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  23. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  24. 2025Why DPO is a Misspecified Estimator and How to Fix Itcited by 3
  25. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modelingcited by 3
  26. 2024Meta-Learning Objectives for Preference Optimizationcited by 2
  27. 2025Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generationcited by 2
  28. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Samplingcited by 1
  29. 2026LLM-HYPER: Generative CTR Modeling for Cold-Start Ad Personalization via LLM-Based Hypernetworkscited by 1
  30. 2026General Preference Reinforcement Learningcited by 1
  31. 2026Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Correctionscited by 1
  32. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  33. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  34. 2025Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficultycited by 0
  35. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  36. 2026Reward Learning through Ranking Mean Squared Errorcited by 0
  37. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  38. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Design Considerations in Offline Preference-based RL
  2. 2024OPTune: Efficient Online Preference Tuning
  3. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  4. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  5. 2024Filtered Direct Preference Optimization
  6. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  7. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  8. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications