mlsanityData to 7 Oct 2026

Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution

arXiv 2602.06239 ↗First posted 5 Feb 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce PEPO (Pessimistic Ensemble based Preference Optimization), a single-step Direct Preference Optimization (DPO)-like algorithm to mitigate the well-known over-optimization issue in preference learning without requiring the knowledge of the data-generating distribution or learning an explicit reward model. PEPO achieves pessimism via an ensemble of preference-optimized policies trained on disjoint data subsets and then aggregates them through a worst case construction that favors the agreement across models. In the tabular setting, PEPO achieves sample complexity guarantees depending only on a single-policy concentrability coefficient, thus avoiding the all-policy concentrability which affects the guarantees of algorithms prone to over-optimization, such as DPO. The theoretical findings are corroborated by a convincing practical performance, while retaining the simplicity and the practicality of DPO-style training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 69 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

60 of 69 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains31 Jan
  2. 2025Robust Preference Optimization via Dynamic Target Margins4 Jun
  3. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts5 Dec
  4. 2024On Extending Direct Preference Optimization to Accommodate Ties25 Sep
  5. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  6. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models6 Jun
  7. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  8. 2024Robust Preference Optimization through Reward Model Distillation29 May
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  10. 2024Token-level Direct Preference Optimization18 Apr
  11. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  12. 2024Learn Your Reference Model for Real Good Alignment15 Apr
4 more
  1. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  2. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 44

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  10. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
32 more
  1. 2021Epistemic Neural Networkscited by 120
  2. 2024To Believe or Not to Believe Your LLMcited by 120
  3. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  7. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  8. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  9. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  10. 2020Near-optimal Regret Bounds for Stochastic Shortest Pathcited by 50
  11. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  12. 2014Near-optimal Reinforcement Learning in Factored MDPscited by 41
  13. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  14. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  15. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  16. 2021Online Apprenticeship Learningcited by 28
  17. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  18. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 17
  19. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  20. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  21. 2024Reward Learning From Preference With Tiescited by 13
  22. 2024More Efficient Randomized Exploration for Reinforcement Learning via Approximate Samplingcited by 12
  23. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learningcited by 12
  24. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  25. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  26. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  27. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Modelscited by 9
  28. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
  29. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  30. 2025Design Considerations in Offline Preference-based RLcited by 4
  31. 2024Batch Ensemble for Variance Dependent Regret in Stochastic Banditscited by 3
  32. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
  2. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  3. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  4. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimization
  5. 2025Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixing
  6. 2025Why DPO is a Misspecified Estimator and How to Fix It
  7. 2026Intrinsic Mutual Information as a Modulator for Preference Optimization
  8. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimization