Maximizing Signal in Human-Model Preference Alignment
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
The emergence of powerful LLMs has led to a paradigm shift in Natural Language Understanding and Natural Language Generation. The properties that make LLMs so valuable for these tasks -- creativity, ability to produce fluent speech, and ability to quickly and effectively abstract information from large corpora -- also present new challenges to evaluating their outputs. The rush to market has led teams to fall back on quick, cost-effective automatic evaluations which offer value, but do not obviate the need for human judgments in model training and evaluation. This paper argues that in cases in which end users need to agree with the decisions made by ML models -- e.g. in toxicity detection or extraction of main points for summarization -- models should be trained and evaluated on data that represent the preferences of those users. We support this argument by explicating the role of human feedback in labeling and judgment tasks for model training and evaluation. First, we propose methods for disentangling noise from signal in labeling tasks. Then we show that noise in labeling disagreement can be minimized by adhering to proven methodological best practices, while signal can be maximized to play an integral role in model training and evaluation tasks. Finally, we illustrate best practices by providing a case study in which two guardrails classifiers are evaluated using human judgments to align final model behavior to user preferences. We aim for this paper to provide researchers and professionals with guidelines to integrating human judgments into their ML and generative AI evaluation toolkit, particularly when working toward achieving accurate and unbiased features that align with users' needs and expectations.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 40 references that are in the map. First posted on arXiv 6 Mar 2025.
Builds on
11 of 40 references.
References · 11
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
- 2019Human uncertainty makes classification more robustcited by 235
- 2017Why We Need New Evaluation Metrics for NLGcited by 233
- 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
- 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
- 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 61
- 2021The Challenge of Value Alignment: from Fairer Algorithms to AI Safetycited by 35
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Dissecting Human and LLM Preferences
- 2025Aligning Black-box Language Models with Human Judgments
- 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
- 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
- 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
- 2025Users as Annotators: LLM Preference Learning from Comparison Mode
- 2026DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
- 2024A Survey on Human Preference Learning for Large Language Models