mlsanityData to 7 Oct 2026

Users as Annotators: LLM Preference Learning from Comparison Mode

arXiv 2510.13830 ↗First posted 10 Oct 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Pairwise preference data have played an important role in the alignment of large language models (LLMs). Each sample of such data consists of a prompt, two different responses to the prompt, and a binary label indicating which of the two responses is better. The labels are usually annotated by professional human annotators. In this paper, we consider an alternative approach to collect pairwise preference data -- user annotation from comparison mode. With the increasingly wider adoption of LLMs among the population, users are contributing more and more of their preference labels through their daily interactions with the LLMs. The upside of such labels is that users are the best experts in judging the responses to their own queries/prompts, but the downside is the lack of quality control in these labels. In this paper, we consider a new idea of generating two responses from two different models or two different versions of the same model. The asymmetry allows us to make an inference of the user's data quality through our proposed user behavior model. We develop an expectation-maximization algorithm to estimate a latent quality factor of the user, and filter users' annotation data accordingly. The downstream task shows the effectiveness of our approach in both capturing the user behavior and data filtering for LLM alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 58 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

39 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators10 Feb
  2. 2024Aligning LLM Agents by Learning Latent Preference from User Edits23 Apr
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec

Other references · 36

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
24 more
  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  2. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  4. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  5. 2014Statistical guarantees for the EM algorithm: From population to sample-based analysiscited by 131
  6. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  7. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  8. 2014Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcingcited by 63
  9. 2017Learning From Noisy Singly-labeled Datacited by 50
  10. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  11. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 39
  12. 2022CICERO: A Dataset for Contextualized Commonsense Inference in Dialoguescited by 38
  13. 2013Crowdsourced Judgement Elicitation with Endogenous Proficiencycited by 30
  14. 2021Annotation Inconsistency and Entity Bias in MultiWOZcited by 30
  15. 2014Optimum Statistical Estimation with Strategic Data Sourcescited by 28
  16. 2020Neural Modeling for Named Entities and Morphology (NEMO^2)cited by 26
  17. 2020Author's Sentiment Predictioncited by 18
  18. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  19. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  20. 2024Incentivizing Quality Text Generation via Statistical Contractscited by 10
  21. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  22. 2024On Efficient and Statistical Quality Estimation for Data Annotationcited by 8
  23. 2023Algorithmic Persuasion Through Simulationcited by 4
  24. 2024LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systemscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  2. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  3. 2024Dissecting Human and LLM Preferences
  4. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  5. 2024Aligning to Thousands of Preferences via System Message Generalization
  6. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  7. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  8. 2024REAL: Response Embedding-based Alignment for LLMs