mlsanityData to 14 Oct 2026

Scalable Valuation of Human Feedback through Provably Robust Model Alignment

arXiv 2505.17859 ↗First posted 23 May 2025cs.LG cs.AI stat.MLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Despite the importance of aligning language models with human preferences, crowd-sourced human feedback is often noisy -- for example, preferring less desirable responses -- posing a fundamental challenge to alignment. A truly robust alignment objective should yield identical model parameters even under severe label noise, a property known as redescending. We prove that no existing alignment methods satisfy this property. To address this, we propose H"older-DPO, the first principled alignment loss with a provable redescending property, enabling estimation of the clean data distribution from noisy feedback. The aligned model estimates the likelihood of clean data, providing a theoretically grounded metric for dataset valuation that identifies the location and fraction of mislabels. This metric is gradient-free, enabling scalable and automated human feedback valuation without costly manual verification or clean validation dataset. H"older-DPO achieves state-of-the-art robust alignment performance while accurately detecting mislabels in controlled datasets. Finally, applied to Anthropic HH-RLHF dataset, it reveals substantial noise levels and removing these mislabels significantly improves alignment performance across methods. The code is available at https://github.com/ma921/HolderDPO.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 61 of its 101 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

61 of 101 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Learning Dynamics of LLM Finetuning15 Jul
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  4. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  5. 2024ROPO: Robust Preference Optimization for Large Language Models5 Apr
  6. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  7. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference5 Dec
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  10. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2020Learning to summarize from human feedbackcited by 2,588
36 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2023Textbooks Are All You Needcited by 569
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  12. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  13. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  14. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  15. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  16. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  17. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  18. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  19. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  20. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
  21. 2024Data Shapley in One Training Runcited by 81
  22. 2021Robust Generalised Bayesian Inference for Intractable Likelihoodscited by 53
  23. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 44
  24. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  25. 2024RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Responsecited by 24
  26. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
  27. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  28. 2024Do Influence Functions Work on Large Language Models?cited by 20
  29. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  30. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 15
  31. 2024How Much Can We Forget about Data Contamination?cited by 14
  32. 2024Delta-Influence: Unlearning Poisons via Influence Functionscited by 13
  33. 2025Detecting and Filtering Unsafe Training Data via Data Attribution with Denoised Representationcited by 9
  34. 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
  35. 2025Bayesian Optimization for Building Social-Influence-Free Consensuscited by 6
  36. 2025On Corruption-Robustness in Performative Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 5
  2. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignmentcited by 2
  3. 2026Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputscited by 1
  4. 2026Generalised Robust Bayes for Joint Inference of Model and Contaminationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024SEAL: Systematic Error Analysis for Value ALignment
  2. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  3. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  4. 2023Aligning Large Language Models through Synthetic Feedback
  5. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  6. 2025Doubly Robust Alignment for Large Language Models
  7. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  8. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases