mlsanityData to 7 Oct 2026

BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment

arXiv 2609.35044 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator's bias up to a shared constant. The remaining constant is what fixes the aligned model's attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability 0.96 and BA-DPO removes 81 to 95% of that shift; on MultiPref with real annotators it removes about half of DPO's lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 37 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

31 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025ADPO: Anchored Direct Preference Optimization21 Oct
  2. 2025A Survey of Direct Preference Optimization12 Mar
  3. 2024Length Desensitization in Direct Preference Optimization10 Sep
  4. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence16 Jun
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences23 May
  7. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  9. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  10. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  11. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  12. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
9 more
  1. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  2. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  3. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  4. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  5. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  6. 2024BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimizationcited by 18
  7. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7
  8. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  9. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
  4. 2026Private Direct Preference Optimization for LLM Alignment
  5. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  6. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  7. 2024sDPO: Don't Use Your Data All at Once
  8. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts