mlsanityData to 7 Oct 2026

Human Alignment of Large Language Models through Online Preference Optimisation

arXiv 2403.08635 ↗First posted 13 Mar 2024cs.LG cs.AI stat.MLcited by 90

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 52 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

42 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2021Convex optimizationcited by 2,479
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
25 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  13. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  14. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  15. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  16. 2023Nash Learning from Human Feedbackcited by 220
  17. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  18. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  19. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  20. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  21. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  22. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  23. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  24. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  25. 2022Reward Gaming in Conditional Text Generationcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 34; 2026: 16.

90 papers.

Sort:

By area:

  1. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  2. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  3. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  4. 2024Generative Reward Modelscited by 148
  5. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  6. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  7. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  8. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  9. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  10. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  11. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  12. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
38 more
  1. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  2. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  3. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  4. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  5. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  6. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  9. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  10. 2025Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learningcited by 27
  11. 2025A Survey of Personalization: From RAG to Agentcited by 26
  12. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  13. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  14. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  15. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  16. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  17. 2024Self-Improving Robust Preference Optimizationcited by 19
  18. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersionscited by 18
  19. 2024SPRec: Self-Play to Debias LLM-based Recommendationcited by 18
  20. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  21. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  22. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  23. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  24. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  25. 2024Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leapscited by 12
  26. 2024Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolutioncited by 11
  27. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11
  28. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  29. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  30. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  31. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  32. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  33. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9
  34. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9
  35. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  36. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 9
  37. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processcited by 8
  38. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  2. 2024Towards Efficient Exact Optimization of Language Model Alignment
  3. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2024Language Alignment via Nash-learning and Adaptive feedback
  6. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  7. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  8. 2024Accelerated Preference Optimization for Large Language Model Alignment