mlsanityData to 14 Oct 2026

Incentivizing High-Quality Human Annotations with Golden Questions

arXiv 2505.19134 ↗First posted 25 May 2025cs.GT cs.LG stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human-annotated data plays a vital role in training large language models (LLMs), such as supervised fine-tuning and human preference alignment. However, it is not guaranteed that paid human annotators produce high-quality data. In this paper, we study how to incentivize human annotators to do so. We start from a principal-agent model to model the dynamics between the company (the principal) and the annotator (the agent), where the principal can only monitor the annotation quality by examining n samples. We investigate the maximum likelihood estimators (MLE) and the corresponding hypothesis testing to incentivize annotators: the agent is given a bonus if the MLE passes the test. By analyzing the variance of the outcome, we show that the strategic behavior of the agent makes the hypothesis testing very different from traditional ones: Unlike the exponential rate proved by the large deviation theory, the principal-agent model's hypothesis testing rate is of Theta(1/√n log n). Our theory implies two criteria for the golden questions to monitor the performance of the annotators: they should be of (1) high certainty and (2) similar format to normal ones. In that light, we select a set of golden questions in human preference data. By doing incentive-compatible experiments, we find out that the annotators' behavior is better revealed by those golden questions, compared to traditional survey techniques such as instructed manipulation checks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 98 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

47 of 98 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators10 Feb
  2. 2020Learning to summarize from human feedback2 Sep

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  10. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
33 more
  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  2. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  3. 2023Nash Learning from Human Feedbackcited by 224
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  5. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  6. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  7. 2018A Marketplace for Data: An Algorithmic Solutioncited by 76
  8. 2014Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problemscited by 46
  9. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 41
  10. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  11. 2022CICERO: A Dataset for Contextualized Commonsense Inference in Dialoguescited by 38
  12. 2018Simple versus Optimal Contractscited by 33
  13. 2013Crowdsourced Judgement Elicitation with Endogenous Proficiencycited by 30
  14. 2021Annotation Inconsistency and Entity Bias in MultiWOZcited by 30
  15. 2014Optimum Statistical Estimation with Strategic Data Sourcescited by 28
  16. 2020Neural Modeling for Named Entities and Morphology (NEMO^2)cited by 26
  17. 2020Author's Sentiment Predictioncited by 18
  18. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  19. 2020The Complexity of Contractscited by 16
  20. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  21. 2022Principal-Agent Hypothesis Testingcited by 14
  22. 2014Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcingcited by 11
  23. 2023Delegated Classificationcited by 10
  24. 2024Incentivizing Quality Text Generation via Statistical Contractscited by 10
  25. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  26. 2024Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contractscited by 9
  27. 2023Delegating Data Collection in Decentralized Machine Learningcited by 8
  28. 2024On Efficient and Statistical Quality Estimation for Data Annotationcited by 8
  29. 2024Algorithmic Contract Theory: A Surveycited by 8
  30. 2023Algorithmic Persuasion Through Simulationcited by 4
  31. 2024Repeated Contracting with Multiple Non-Myopic Agents: Policy Regret and Limited Liabilitycited by 4
  32. 2024LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systemscited by 4
  33. 2023Monitoring with Rich Datacited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Large Language Models as Expert Annotators
  2. 2025Users as Annotators: LLM Preference Learning from Comparison Mode
  3. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
  4. 2025MAFA: A multi-agent framework for annotation
  5. 2025Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
  7. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
  8. 2024Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework