Reliability-Aware LLM Alignment from Inconsistent Human Feedback
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences. However, its efficacy is often compromised by the inherent inconsistency and subjectivity of human annotations. Existing preference optimization frameworks, such as Direct Preference Optimization (DPO), typically treat ambiguous pairs with high annotator disagreement identically to those with unanimous consensus, forcing models to overfit to inconsistent supervision signals and leading to suboptimal alignment. In this work, we propose Reliability-Guided Preference Optimization (RGPO), a robust framework designed to mitigate the impact of inconsistent human feedback. RGPO estimates annotator reliability and infers latent ground truth labels from noisy human feedback to identify robust preferences. Furthermore, we introduce a reliability-aware consistency optimization that dynamically modulates the training objective based on the consensus level of annotations, ensuring the model prioritizes high-consensus supervision signals. Extensive experiments on LLM alignment benchmarks demonstrate that RGPO effectively reduces inconsistency and noise in training data and achieves superior performance compared to widely adopted RLHF baselines. Our code and configurations are available at https://github.com/GenieHuang/RGPO.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 47 references that are in the map. First posted on arXiv 7 Jul 2026.
Builds on
40 of 47 references.
Closest ancestors · 9
Same direction, newest first
- 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
- 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
- 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
- 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
- 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
- 2023Preference Ranking Optimization for Human Alignment30 Jun
- 2022Training language models to follow instructions with human feedback4 Mar
- 2021A General Language Assistant as a Laboratory for Alignment1 Dec
Other references · 31
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2020Learning to summarize from human feedbackcited by 2,562
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
19 more
- 2023Instruction Tuning with GPT-4cited by 676
- 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
- 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
- 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
- 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
- 2022Teaching language models to support answers with verified quotescited by 290
- 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
- 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
- 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
- 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
- 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
- 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
- 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
- 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
- 2019Outlier Detection for Improved Data Quality and Diversity in Dialog Systemscited by 28
- 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
- 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
- 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
- 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
- 2025RLTHF: Targeted Human Feedback for LLM Alignment
- 2025Doubly Robust Alignment for Large Language Models
- 2024Towards Efficient Exact Optimization of Language Model Alignment
- 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment