mlsanityData to 14 Oct 2026

Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment

arXiv 2509.23564 ↗First posted 28 Sep 2025cs.AI cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Human feedback plays a pivotal role in aligning large language models (LLMs) with human preferences. However, such feedback is often noisy or inconsistent, which can degrade the quality of reward models and hinder alignment. While various automated data cleaning methods have been proposed to mitigate this issue, a systematic evaluation of their effectiveness and generalizability remains lacking. To bridge this gap, we introduce the first comprehensive benchmark for evaluating 13 preference data cleaning methods in the context of LLM alignment. PrefCleanBench offers a standardized protocol to assess cleaning strategies in terms of alignment performance and generalizability across diverse datasets, model architectures, and optimization algorithms. By unifying disparate methods and rigorously comparing them, we uncover key factors that determine the success of data cleaning in alignment tasks. This benchmark lays the groundwork for principled and reproducible approaches to improving LLM alignment through better data quality-highlighting the crucial but underexplored role of data preprocessing in responsible AI development. We release modular implementations of all methods to catalyze further research: https://github.com/deeplearning-wisc/PrefCleanBench.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 83 of its 94 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

83 of 94 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment13 Sep
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024Distributional Preference Alignment of LLMs via Optimal Transport9 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Token-level Direct Preference Optimization18 Apr
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  7. 2024Understanding the Learning Dynamics of Alignment with Human Feedback27 Mar
  8. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  9. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  10. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  11. 2024Unintended Impacts of LLM Alignment on Global Representation22 Feb
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
10 more
  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  2. 2024Active Preference Learning for Large Language Models12 Feb
  3. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  4. 2024Decoding-time Realignment of Language Models5 Feb
  5. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback21 Jan
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  9. 2023Preference Ranking Optimization for Human Alignment30 Jun
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 61

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023Mistral 7Bcited by 3,164
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2020Learning to summarize from human feedbackcited by 2,588
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  8. 2023Whose Opinions Do Language Models Reflect?cited by 773
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2023Textbooks Are All You Needcited by 569
  13. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  14. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  15. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  16. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  17. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  18. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  19. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  20. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  21. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  22. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  23. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  24. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  25. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  26. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  27. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  28. 2023Nash Learning from Human Feedbackcited by 224
  29. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  30. 2023Model evaluation for extreme riskscited by 177
  31. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  32. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  33. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  34. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  35. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  36. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  37. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  38. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 4
  2. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  2. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment
  3. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  5. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2025ComPO: Preference Alignment via Comparison Oracles