mlsanityData to 14 Oct 2026

Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

arXiv 2603.21404 ↗First posted 22 Mar 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 73 references that are in the map. First posted on arXiv 22 Mar 2026.

Builds on

31 of 73 references.

References · 31

  1. 2023Whose Opinions Do Language Models Reflect?cited by 773
  2. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  3. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  4. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  5. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  6. 2024Can LLM be a Personalized Judge?cited by 100
  7. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  8. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  9. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  10. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
  11. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
  12. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
19 more
  1. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 52
  2. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  3. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  4. 2021Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detectioncited by 38
  5. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  6. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 33
  7. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  8. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  9. 2024Enhancing Text Classification through LLM-Driven Active Learning and Human Annotationcited by 20
  10. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 18
  11. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  12. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 16
  13. 2024Counterspeakers' Perspectives: Unveiling Barriers and AI Needs in the Fight against Online Hatecited by 15
  14. 2026Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLPcited by 13
  15. 2021Do Natural Language Explanations Represent Valid Logical Arguments? Verifying Entailment in Explainable NLI Gold Standardscited by 12
  16. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  17. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  18. 2025Missing the Margins: A Systematic Literature Review on the Demographic Representativeness of LLMscited by 9
  19. 2023Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Trainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Cluster-Robust Prediction-Powered Inferencecited by 0