mlsanityData to 14 Oct 2026

Leveraging Robust Optimization for LLM Alignment under Distribution Shifts

arXiv 2504.05831 ↗First posted 8 Apr 2025cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference alignment methods are increasingly critical for steering large language models (LLMs) to generate outputs consistent with human values. While recent approaches often rely on synthetic data generated by LLMs for scalability and cost-efficiency reasons, this reliance can introduce distribution shifts that undermine the nuanced representation of human preferences needed for desirable outputs. In this paper, we propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts. Our approach first leverages well-learned classifiers to assign a calibration value to each training sample, quantifying its alignment with the target human-preferred distribution. These values are then incorporated into a robust optimization objective that minimizes the worst-case loss over regions of the data space most relevant to human preferences. By explicitly focusing optimization on the target distribution, our approach mitigates the impact of distributional mismatch and improves the generation of responses that better reflect intended values.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 58 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

39 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization5 Sep
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey4 Sep
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024LIRE: listwise reward enhancement for preference alignment22 May
  6. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  7. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  8. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  9. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  10. 2024Towards Efficient Exact Optimization of Language Model Alignment1 Feb
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
2 more
  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  12. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
13 more
  1. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  2. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  3. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  4. 2023The False Promise of Imitating Proprietary LLMscited by 263
  5. 2019Distributionally Robust Language Modelingcited by 143
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  7. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  8. 2021DORO: Distributional and Outlier Robust Optimizationcited by 56
  9. 2024Improving Reward Models with Synthetic Critiquescited by 44
  10. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  11. 2017Boosted Generative Modelscited by 18
  12. 2020Stable Adversarial Learning under Distributional Shiftscited by 16
  13. 2023Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implicationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 4
  2. 2026Structured Credal Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  2. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  3. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  4. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  5. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  6. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  7. 2026Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning
  8. 2025ComPO: Preference Alignment via Comparison Oracles