mlsanityData to 7 Oct 2026

Uncertainty-Normalized Margins for Direct Preference Optimization

arXiv 2609.38647 ↗First posted 29 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods also fits: Reward modeling and RLHF

Abstract (arXiv)

Direct preference optimization (DPO) models binary preferences through a Bradley-Terry model with a common noise scale, without explicitly accounting for preference strength or prompt-dependent uncertainty from human feedback. We introduce uncertainty-normalized margin DPO (UNM-DPO), which combines strength-dependent margins with a learned prompt scale. Motivated by a heteroskedastic Bradley-Terry model, we develop two training objectives. Both compare the implicit rewards of preferred and rejected responses, derived from response log-probability ratios to a reference policy. Advantage-only (AO) divides this reward difference by the prompt scale before subtracting the margin; whole-residual (WR) subtracts the margin before dividing by the scale. For the WR comparison model, we establish a necessary and sufficient condition under which known margins make the prompt scale identifiable. We introduce a practical procedure for learning the scale. Building on WR, we introduce ULNM-DPO-WR, which normalizes each response's implicit reward by its length. We evaluate our methods against DPO and related baselines on HelpSteer2 and HelpSteer3, using the Skywork reward model as a judge. With Llama-3.1-8B-Instruct, ULNM-DPO-WR achieves tie-adjusted win rates against matched DPO of 68.00% and 65.31% on evaluation panels, with higher mean rewards and shorter responses on average. On AlpacaEval with a GPT-4.1 judge and GPT-4-Turbo reference answers, the same 8B policy achieves a length-controlled win rate of 21.62%, compared with 16.39% for DPO and 15.30% for SimPO. These results demonstrate the potential of combining preference-strength margins, learned prompt scales, and length normalization for policy optimization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 23 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

18 of 23 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment7 Jan
  2. 2025Robust Preference Optimization via Dynamic Target Margins4 Jun
  3. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback4 Oct
  4. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributions30 Apr
  7. 2024Direct Preference Optimization with an Offset16 Feb
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 10

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  7. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  8. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  9. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  10. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
  2. 2026xi-DPO: Direct Preference Optimization via Ratio Reward Margin
  3. 2025Length-Controlled Margin-Based Preference Optimization without Reference Model
  4. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  5. 2025Adaptive Margin RLHF via Preference over Preferences
  6. 2026Boosting Direct Preference Optimization with Penalization
  7. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  8. 2026Autoregressive Direct Preference Optimization