mlsanityData to 7 Oct 2026

Towards Efficient Exact Optimization of Language Model Alignment

arXiv 2402.00856 ↗First posted 1 Feb 2024cs.CLcited by 37

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The alignment of language models with human preferences is vital for their application in real-world tasks. The problem is formulated as optimizing the model's policy to maximize the expected reward that reflects human preferences with minimal deviation from the initial policy. While considered as a straightforward solution, reinforcement learning (RL) suffers from high variance in policy updates, which impedes efficient policy improvement. Recently, direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. However, we show that DPO derived based on the optimal solution of the problem leads to a compromised mean-seeking approximation of the optimal solution in practice. In this paper, we propose efficient exact optimization (EXO) of the alignment objective. EXO is guaranteed to optimize in the same direction as RL algorithms asymptotically for arbitrary policy parametrization. This leads to the same mode-seeking solution, while enables efficient optimization by circumventing the complexities of RL. We also compare our method to DPO with both theoretical and empirical analyses, and further demonstrate the advantages of our method over existing approaches on realistic human preference data. Code is available at https://github.com/haozheji/exact-optimization.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

38 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
21 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  7. 2020Critic Regularized Regressioncited by 269
  8. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  9. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  10. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  11. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  12. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  13. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  14. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  15. 2020Limitations of Autoregressive Models and Their Alternativescited by 54
  16. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48
  17. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  18. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  19. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  20. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  21. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 23; 2026: 6.

37 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  5. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  6. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  7. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  8. 2024PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinkingcited by 17
  9. 2024SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimizationcited by 14
  10. 2025In-situ graph reasoning and knowledge expansion using Graph-PReFLexORcited by 11
  11. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  12. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 9
25 more
  1. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignmentcited by 8
  2. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  3. 2025Inference-time Alignment in Continuous Spacecited by 7
  4. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  5. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  6. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modelingcited by 4
  7. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3
  8. 2025Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learningcited by 3
  9. 2025MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignmentcited by 3
  10. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
  11. 2024The Superalignment of Superhuman Intelligence with Large Language Modelscited by 2
  12. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shiftscited by 2
  13. 2026When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropycited by 2
  14. 2025Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Modelscited by 1
  15. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
  16. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Samplingcited by 1
  17. 2025Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraintscited by 1
  18. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?cited by 1
  19. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  20. 2026FlowCPO: A Unified Divergence View of Preference Alignment for Flow Modelscited by 1
  21. 2025Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimatorcited by 0
  22. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
  23. 2026Towards Precision Therapy in Hepatocellular Carcinoma: A Clinical-Reasoning LLM for Risk Stratification and Treatment Guidancecited by 0
  24. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillationcited by 0
  25. 2026Direct Preference Density Alignment for Conversational Audio Equalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2025A Principled Loss Function for Direct Language Model Alignment
  3. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  4. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  6. 2024Accelerated Preference Optimization for Large Language Model Alignment
  7. 2026A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
  8. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment