mlsanityData to 7 Oct 2026

Preference Robustness for DPO with Applications to Public Health

arXiv 2509.02709 ↗First posted 2 Sep 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study an LLM fine-tuning task for designing reward functions for sequential resource allocation problems in public health, guided by human preferences expressed in natural language. This setting presents a challenging testbed for alignment due to complex and ambiguous objectives and limited data availability. We propose DPO-PRO, a robust fine-tuning algorithm based on Direct Preference Optimization (DPO), which accounts for uncertainty in the preference distribution using a lightweight Distributionally Robust Optimization (DRO) formulation. Unlike prior DRO-based DPO methods, DPO-PRO is significantly less conservative. We evaluate DPO-PRO on a real-world maternal mobile health program operated by the non-profit organization ARMMAN, as well as on standard alignment benchmarks. Experimental results demonstrate that our method consistently improves robustness to noisy preference signals compared to existing DPO variants. Moreover, DPO-PRO achieves comparable performance to prior self-reflection-based baseline for reward function design, while requiring significantly lower inference-time cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 8 of its 45 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

8 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization4 Feb
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications21 Oct
  3. 2024Orthogonal Finetuning for Direct Preference Optimization23 Sep

Other references · 5

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  3. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  4. 2024Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewardscited by 9
  5. 2025Towards Understanding Why Data Augmentation Improves Generalizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Lightweight Robust Direct Preference Optimization
  2. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
  3. 2024Minor DPO reject penalty to increase training robustness
  4. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  5. 2024Direct Preference Optimization with an Offset
  6. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  7. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  8. 2025A Survey of Direct Preference Optimization