mlsanityData to 14 Oct 2026

Offline Regularised Reinforcement Learning for Large Language Models Alignment

arXiv 2405.19107 ↗First posted 29 May 2024cs.LG cs.AIcited by 48

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The dominant framework for alignment of large language models (LLM), whether through reinforcement learning from human feedback or direct preference optimisation, is to learn from preference data. This involves building datasets where each element is a quadruplet composed of a prompt, two independent responses (completions of the prompt) and a human preference between the two independent responses, yielding a preferred and a dis-preferred response. Such data is typically scarce and expensive to collect. On the other hand, single-trajectory datasets where each element is a triplet composed of a prompt, a response and a human feedback is naturally more abundant. The canonical element of such datasets is for instance an LLM's response to a user's prompt followed by a user's feedback such as a thumbs-up/down. Consequently, in this work, we propose DRO, or Direct Reward Optimisation, as a framework and associated algorithms that do not require pairwise preferences. DRO uses a simple mean-squared objective that can be implemented in various ways. We validate our findings empirically, using T5 encoder-decoder language models, and show DRO's performance over selected baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that DRO is a simple and empirically compelling method for single-trajectory policy optimisation.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 63 of its 80 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

63 of 80 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  2. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization31 Mar
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  4. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization4 Mar
  5. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2024Towards Efficient Exact Optimization of Language Model Alignment1 Feb
  8. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  9. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  12. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
38 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  13. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  16. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  17. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  18. 2022Self-critiquing models for assisting human evaluatorscited by 380
  19. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  20. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  21. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  22. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  23. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  24. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  25. 2023Nash Learning from Human Feedbackcited by 224
  26. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  27. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  28. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  29. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  30. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  31. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  32. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  33. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  34. 2021Consequences of Misaligned AIcited by 80
  35. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  36. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  37. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 44
  38. 2022Reward Gaming in Conditional Text Generationcited by 30

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 26; 2026: 10.

48 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 104
  3. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  4. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  5. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  6. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  7. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  8. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  9. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  10. 2025A Survey of Direct Preference Optimizationcited by 36
  11. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  12. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
36 more
  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 30
  2. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  3. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  4. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  5. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  6. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  7. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimizationcited by 17
  8. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  9. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  10. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11
  11. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  12. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  13. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10
  14. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  15. 2024UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Functioncited by 9
  16. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  17. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8
  18. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithmscited by 8
  19. 2025Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Predictioncited by 8
  20. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codescited by 7
  21. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  22. 2025Training Language Model to Critique for Better Refinementcited by 7
  23. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Modelscited by 7
  24. 2024UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Typescited by 6
  25. 2025EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructionscited by 6
  26. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
  27. 2025ShiQ: Bringing back Bellman to LLMscited by 3
  28. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  29. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Managementcited by 1
  30. 2025Value-Free Policy Optimization via Reward Partitioningcited by 1
  31. 2026rePIRL: Learn PRM with Inverse RL for LLM Reasoningcited by 1
  32. 2026Approximation of Log-Partition Function in Policy Mirror Descent Induces Implicit Regularization for LLM Post-Trainingcited by 0
  33. 2026Causal Direct Preference Optimization for Distributionally Robust Generative Recommendationcited by 0
  34. 2026CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learningcited by 0
  35. 2026Toward a Theory of Value in AI Alignmentcited by 0
  36. 2026RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  4. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  5. 2025A Principled Loss Function for Direct Language Model Alignment
  6. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  7. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  8. 2024Investigating on RLHF methodology