mlsanityData to 14 Oct 2026

On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

arXiv 2505.24709 ↗First posted 30 May 2025cs.LG cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Optimizing policies based on human preferences is key to aligning language models with human intent. This work focuses on reward modeling, a core component in reinforcement learning from human feedback (RLHF), and offline preference optimization, such as direct preference optimization. Conventional approaches typically assume accurate annotations. However, real-world preference data often contains noise due to human errors or biases. We propose a principled framework for robust policy optimization under noisy preferences, viewing reward modeling as a classification problem. This allows us to leverage symmetric losses, known for their robustness to label noise in classification, leading to our Symmetric Preference Optimization (SymPO) method. We prove that symmetric losses enable successful policy optimization even under noisy labels, as the resulting reward remains rank-preserving -- a property sufficient for policy improvement. Experiments on synthetic and real-world tasks demonstrate the effectiveness of SymPO.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 43 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

27 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  10. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  11. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  12. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
12 more
  1. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  2. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  3. 2016Variance-based regularization with convex objectivescited by 155
  4. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  5. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  6. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  7. 2019On Symmetric Losses for Learning from Corrupted Labelscited by 53
  8. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 53
  9. 2018On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Datacited by 43
  10. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  11. 2019Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approachcited by 29
  12. 2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbackscited by 7
  2. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweightingcited by 6
  3. 2026Mitigating Reward Hacking in RLHF via Advantage Sign Robustnesscited by 1
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024Robust Reinforcement Learning from Corrupted Human Feedback
  3. 2025Mitigating Preference Hacking in Policy Optimization with Pessimism
  4. 2026Optimal Transport for LLM Reward Modeling from Noisy Preference
  5. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  6. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  8. 2024Towards Reliable Alignment: Uncertainty-aware RLHF