mlsanityData to 7 Oct 2026

Distributionally Robust Listwise Preference Optimization

arXiv 2607.01715 ↗First posted 2 Jul 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from K! enumeration to O(Klog K). This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global ε-suboptimality with O(ε-2) sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak convexity and widetilde O(ε-2) Moreau-envelope stationarity. Experiments in offline LLM alignment show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise. In online alignment, it makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 24 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

17 of 24 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Oracle-Robust Online Alignment for Large Language Models24 Feb
  2. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization4 Feb
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  4. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models21 Jun
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  7. 2023Preference Ranking Optimization for Human Alignment30 Jun
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  6. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  7. 2016Error bounds, quadratic growth, and linear convergence of proximal methodscited by 59
  8. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  9. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models
  2. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  3. 2024ROPO: Robust Preference Optimization for Large Language Models
  4. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
  5. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  6. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  7. 2026Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment
  8. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling