mlsanityData to 14 Oct 2026

One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise

arXiv 2503.12301 ↗First posted 16 Mar 2025cs.LG cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) have made significant strides in generating human-like responses, largely due to preference alignment techniques. However, these methods often assume unbiased human feedback, which is rarely the case in real-world scenarios. This paper introduces Content-Aware Noise-Resilient Preference Optimization (CNRPO), a novel framework that addresses multiple sources of content-dependent noise in preference learning. CNRPO employs a multi-objective optimization approach to separate true preferences from content-aware noises, effectively mitigating their impact. We leverage backdoor attack mechanisms to efficiently learn and control various noise sources within a single model. Theoretical analysis and extensive experiments on different synthetic noisy datasets demonstrate that CNRPO significantly improves alignment with primary human preferences while controlling for secondary noises and biases, such as response length and harmfulness.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 46 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

28 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence16 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  4. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  12. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
8 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  3. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
  4. 2018Categorizing Variants of Goodhart's Lawcited by 149
  5. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 128
  6. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
  7. 2024Following Length Constraints in Instructionscited by 46
  8. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedbackcited by 3
  2. 2026Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majoritiescited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
  2. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?
  3. 2026Unbiased Alignment for Large Language Models with Noisy Preferences
  4. 2024ROPO: Robust Preference Optimization for Large Language Models
  5. 2025Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data
  6. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
  7. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  8. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment