mlsanityData to 7 Oct 2026

Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives

arXiv 2508.02853 ↗First posted 4 Aug 2025cs.CLcited by 10

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present an approach to modeling annotator disagreement in subjective NLP tasks through both architectural and data-centric innovations. Our model, DEM-MoE (Demographic-Aware Mixture of Experts), routes inputs to expert subnetworks based on annotator demographics, enabling it to better represent structured, group-level variation compared to prior models. DEM-MoE consistently performs competitively across demographic groups, and shows especially strong results on datasets with high annotator disagreement. To address sparse demographic coverage, we test whether LLM-generated synthetic annotations via zero-shot persona prompting can be used for data imputation. We show these synthetic judgments align moderately well with human annotations on our data and offer a scalable way to potentially enrich training data. We then propose and evaluate approaches for blending real and synthetic data using strategies tailored to dataset structure. We find that the optimal strategies depend on dataset structure. Together, these contributions improve the representation of diverse perspectives.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 65 of its 172 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

65 of 172 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Advancing Expert Specialization for Better MoE28 May
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  8. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  9. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  10. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  11. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  12. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
38 more
  1. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  2. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  3. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  4. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  5. 2014Word Representations via Gaussian Embeddingcited by 96
  6. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  7. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  8. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  9. 2021Benchmarking Differentially Private Synthetic Data Generation Algorithmscited by 80
  10. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  11. 2021Designing Toxic Content Classification for a Diversity of Perspectivescited by 63
  12. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  13. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  14. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 51
  15. 2021Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Datacited by 48
  16. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  17. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  18. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  19. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  20. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 31
  21. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  22. 2019Grammatical Error Correction in Low-Resource Scenarioscited by 28
  23. 2023Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Taskscited by 28
  24. 2020Data Weighted Training Strategies for Grammatical Error Correctioncited by 26
  25. 2023Disentangling Perceptions of Offensiveness: Cultural and Moral Correlatescited by 26
  26. 2020Overcoming Data Sparsity in Group Recommendationcited by 22
  27. 2020Balancing Objectives in Counseling Conversations: Advancing Forwards or Looking Backwardscited by 18
  28. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMscited by 14
  29. 2025Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoralcited by 14
  30. 2021Enhanced Speaker-aware Multi-party Multi-turn Dialogue Comprehensioncited by 13
  31. 2024AlignGroup: Learning and Aligning Group Consensus with Member Preferences for Group Recommendationcited by 13
  32. 2023ConsRec: Learning Consensus Behind Interactions for Group Recommendationcited by 11
  33. 2020Towards an Automated SOAP Note: Classifying Utterances from Medical Conversationscited by 10
  34. 2024Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Datacited by 10
  35. 2023Post-processing Private Synthetic Data for Improving Utility on Selected Measurescited by 9
  36. 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
  37. 2018A Dual-Attention Hierarchical Recurrent Neural Network for Dialogue Act Classificationcited by 7
  38. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applicationscited by 7

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluationcited by 21
  2. 2026Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLPcited by 10
  3. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  4. 2026STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systemscited by 3
  5. 2026Multi-Perspective LLM Annotations for Valid Analyses in Subjective Taskscited by 1
  6. 2026Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEMcited by 0
  7. 2026When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detectioncited by 0
  8. 2026Aristotelian Virtue Profiling of LLMs through Ethical Dilemmascited by 0
  9. 2026LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurtscited by 0
  10. 2026Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applicationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement
  2. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators
  3. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
  4. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
  5. 2025Users as Annotators: LLM Preference Learning from Comparison Mode
  6. 2025Value Profiles for Encoding Human Variation
  7. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting
  8. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference