mlsanityData to 14 Oct 2026

Doubly Robust Alignment for Large Language Models

arXiv 2506.01183 ↗First posted 1 Jun 2025cs.LG cs.AI stat.MLcited by 12

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 104 of its 171 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

104 of 171 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025On a Connection Between Imitation Learning and RLHF7 Mar
  2. 2024Understanding Reference Policies in Direct Preference Optimization18 Jul
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  4. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment18 Jun
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  6. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  7. 2024ROPO: Robust Preference Optimization for Large Language Models5 Apr
  8. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  9. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  10. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  11. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
6 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Aligning Language Models with Human Preferences via a Bayesian Approach9 Oct
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  4. 2023Preference Ranking Optimization for Human Alignment30 Jun
  5. 2022Training language models to follow instructions with human feedback4 Mar
  6. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 86

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
38 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  2. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 600
  3. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 478
  4. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 332
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  6. 2019Adapting Neural Networks for the Estimation of Treatment Effectscited by 287
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  8. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  9. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  11. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  12. 2023Nash Learning from Human Feedbackcited by 224
  13. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  14. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  15. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  16. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  17. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  18. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  19. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  20. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  21. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  22. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  23. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  24. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  25. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  26. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  27. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  28. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  29. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  30. 2015Nonparametric methods for doubly robust estimation of continuous treatment effectscited by 64
  31. 2020Semiparametric proximal causal inferencecited by 59
  32. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 58
  33. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  34. 2018Orthogonal Random Forest for Causal Inferencecited by 51
  35. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  36. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
  37. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  38. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  2. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  3. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedbackcited by 9
  4. 2026A Statistical Framework for Alignment with Biased AI Feedbackcited by 4
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  6. 2026LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiencycited by 3
  7. 2025Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Modelcited by 2
  8. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  9. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  10. 2026Learning Perturbations to Extrapolate Your LLMcited by 0
  11. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0
  12. 2026OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  2. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  3. 2025A Principled Loss Function for Direct Language Model Alignment
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2024Investigating on RLHF methodology
  6. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  7. 2025Online Preference Alignment for Language Models via Count-based Exploration
  8. 2024Accelerated Preference Optimization for Large Language Model Alignment