mlsanityData to 7 Oct 2026

Understanding the performance gap between online and offline alignment algorithms

arXiv 2405.08448 ↗First posted 14 May 2024cs.LG cs.AI stat.MLcited by 110

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is the canonical framework for large language model alignment. However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF. Within the context of reward over-optimization, we start with an opening set of experiments that demonstrate the clear advantage of online methods over offline methods. This prompts us to investigate the causes to the performance discrepancy through a series of carefully designed experimental ablations. We show empirically that hypotheses such as offline data coverage and data quality by itself cannot convincingly explain the performance difference. We also find that while offline algorithms train policy to become good at pairwise classification, it is worse at generations; in the meantime the policies trained by online algorithms are good at generations while worse at pairwise classification. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Lastly, we observe that the performance discrepancy persists for both contrastive and non-contrastive loss functions, and appears not to be addressed by simply scaling up policy networks. Taken together, our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

33 of 40 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  4. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  5. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  6. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
13 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  9. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  10. 2023Nash Learning from Human Feedbackcited by 220
  11. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  12. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  13. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 46; 2026: 20.

110 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  3. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  4. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  5. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  6. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  7. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  8. 2024Learning Dynamics of LLM Finetuningcited by 93
  9. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  10. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  11. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  12. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
38 more
  1. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  2. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  3. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  4. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  5. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  6. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  7. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  8. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  9. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  10. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  11. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  12. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  13. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularizationcited by 40
  14. 2025A Survey of Direct Preference Optimizationcited by 35
  15. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  16. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  17. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  18. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  19. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  20. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  21. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  22. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  23. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  24. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Studycited by 22
  25. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 22
  26. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  27. 2024Rational Metareasoning for Large Language Modelscited by 20
  28. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  29. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  30. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  31. 2024Sample-Efficient Alignment for LLMscited by 16
  32. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  33. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  34. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  35. 2025Rethinking DPO: The Role of Rejected Responses in Preference Misalignmentcited by 14
  36. 2025Reinforcement Learning for Flow-Matching Policiescited by 14
  37. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  38. 2025Perception in Reflectioncited by 13

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  4. 2025Design Considerations in Offline Preference-based RL
  5. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  6. 2023Aligning Language Models with Offline Learning from Human Feedback
  7. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  8. 2025Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions