mlsanityData to 7 Oct 2026

From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models

arXiv 2601.06108 ↗First posted 3 Jan 2026cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives -- Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others -- has left practitioners without clear guidance on method selection. This survey provides a textittheoretical unification of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: (I) Preference Model (what likelihood model underlies the objective), (II) Regularization Mechanism (how deviation from reference policies is controlled), and (III) Data Distribution (online vs. offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes -- length hacking, mode collapse, likelihood displacement -- arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 62 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

51 of 62 references.

Closest ancestors · 28

Same direction, newest first

  1. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?1 Oct
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment2 Jun
  3. 2025Doubly Robust Alignment for Large Language Models1 Jun
  4. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration13 Dec
  5. 2024Uncertainty-Penalized Direct Preference Optimization26 Oct
  6. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization5 Sep
  7. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  8. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  9. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  10. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward11 Jun
  11. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  12. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
16 more
  1. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  2. 2024Multi-Reference Preference Optimization for Large Language Models26 May
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling21 May
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  7. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  10. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  11. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  12. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration1 Dec
  13. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  14. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  15. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  16. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  12. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
11 more
  1. 2023Nash Learning from Human Feedbackcited by 220
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  3. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  4. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  5. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  6. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  7. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
  8. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  9. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  10. 2025Design Considerations in Offline Preference-based RLcited by 4
  11. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferencescited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2024Investigating on RLHF methodology
  3. 2024Accelerated Preference Optimization for Large Language Model Alignment
  4. 2024Towards Efficient Exact Optimization of Language Model Alignment
  5. 2025A Principled Loss Function for Direct Language Model Alignment
  6. 2025Online Preference Alignment for Language Models via Count-based Exploration
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization