mlsanityData to 14 Oct 2026

Online Distributionally Robust LLM Alignment via Regression to Relative Reward

arXiv 2509.19104 ↗First posted 23 Sep 2025cs.LG stat.MLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) has become crucial for aligning Large Language Models (LLMs) with human intent. However, existing offline RLHF approaches suffer from overoptimization, where language models degrade by overfitting inaccuracies and drifting from preferred behaviors observed during training. Distributionally robust optimization (DRO) is a natural solution, but existing DRO-DPO methods are sample-inefficient, ignore heterogeneous preferences, and lean on brittle heuristics. We introduce DRO-REBEL, a family of robust online REBEL updates built on type-p Wasserstein, Kullback-Leibler (KL), and chi2 ambiguity sets. Strong duality reduces each update to a relative-reward regression, retaining REBEL's scalability without PPO-style clipping or value networks. Under linear rewards, log-linear policies, and a standard coverage condition, we prove widetildeO(√d/n) bounds on squared parameter error, with sharper constants than prior DRO-DPO analyses, and give the first parametric widetildeO(d/n) rate for DRO-based alignment under preference shift, matching non-robust RLHF in benign regimes. Each divergence yields a tractable SGD-based algorithm: gradient regularization for Wasserstein, importance weighting for KL, and a 1-D dual solve for chi2. On Emotion Alignment, the ArmoRM multi-objective benchmark, and HH-Alignment, DRO-REBEL outperforms prior robust and non-robust baselines across unseen preference mixtures, model sizes, and dataset scales.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 70 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

18 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  2. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  4. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar

Other references · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  7. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  9. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  10. 2024Reward-Robust RLHF in LLMscited by 32
  11. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  12. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shiftcited by 18
2 more
  1. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  2. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Injecting Distributional Awareness into MLLMs via Reinforcement Learning for Deep Imbalanced Regressioncited by 0
  2. 2026Optimal Transport Meets Reinforcement Learning: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
  2. 2025Doubly Robust Alignment for Large Language Models
  3. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  4. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dual
  5. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  6. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  7. 2025A Principled Loss Function for Direct Language Model Alignment
  8. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization