mlsanityData to 14 Oct 2026

Can Large Language Models Capture Dissenting Human Voices?

arXiv 2305.13788 ↗First posted 23 May 2023cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of natural language inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human disagreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to a larger human population. The source code for the experiments is available at https://github.com/xfactlab/emnlp2023-LLM-Disagreement

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 33 of its 66 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

33 of 66 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,856
21 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  7. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  8. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  9. 2022UL2: Unifying Language Learning Paradigmscited by 295
  10. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  11. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  12. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  13. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  14. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  15. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  16. 2021AmbiFC: Fact-Checking Ambiguous Claims with Evidencecited by 38
  17. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  18. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  19. 2022Task Ambiguity in Humans and Language Modelscited by 28
  20. 2021Embracing Ambiguity: Shifting the Training Target of NLI Modelscited by 23
  21. 2023Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 19; 2025: 4; 2026: 7.

33 papers.

Sort:

By area:

  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  2. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  3. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  4. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  5. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  6. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  7. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 26
  8. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  9. 2024"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?cited by 17
  10. 2026Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLPcited by 13
  11. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  12. 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
21 more
  1. 2024A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLIcited by 9
  2. 2025Training and Evaluating with Human Label Variation: An Empirical Studycited by 8
  3. 2025From Evidence to Belief: A Bayesian Epistemology Approach to Language Modelscited by 6
  4. 2024United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Electionscited by 5
  5. 2024Predict the Next Word: Humans exhibit uncertainty in this task and language models _____cited by 4
  6. 2024How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamicscited by 4
  7. 2024Understanding The Effect Of Temperature On Alignment With Human Opinionscited by 4
  8. 2024Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detectioncited by 3
  9. 2025Embracing Diversity: A Multi-Perspective Approach with Soft Labelscited by 3
  10. 2026SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMscited by 2
  11. 2024What's Mine becomes Yours: Defining, Annotating and Detecting Context-Dependent Paraphrases in News Interview Dialogscited by 1
  12. 2024Unraveling the Dominance of Large Language Models Over Transformer Models for Bangla Natural Language Inference: A Comprehensive Studycited by 1
  13. 2024Self-Knowledge Distillation for Learning Ambiguitycited by 1
  14. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  15. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  16. 2024Large Language Models for Integrating Social Determinant of Health Data: A Case Study on Heart Failure 30-Day Readmission Predictioncited by 0
  17. 2024SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environmentcited by 0
  18. 2026Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLPcited by 0
  19. 2026How Ethos and Pathos Appeals Resonate in Reader Interpretations of Social Media Messagescited by 0
  20. 2026Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Modelscited by 0
  21. 2026Narrowing the Horizon: Quantifying Topic Saliency Shifts in Generative Monoculturecited by 0