mlsanityData to 7 Oct 2026

Unbiased Alignment for Large Language Models with Noisy Preferences

arXiv 2607.03248 ↗First posted 3 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 37 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

27 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  2. 2024Discovering Preference Optimization Algorithms with and for Large Language Models12 Jun
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
7 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  5. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  7. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards
  3. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignment
  4. 2025Doubly Robust Alignment for Large Language Models
  5. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  6. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
  7. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  8. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment