mlsanityData to 7 Oct 2026

We're Afraid Language Models Aren't Modeling Ambiguity

arXiv 2304.14399 ↗First posted 27 Apr 2023cs.CLcited by 124

Not on the map. Why?

Abstract (arXiv)

Ambiguity is an intrinsic feature of natural language. Managing ambiguity is a key part of human language understanding, allowing us to anticipate misunderstanding as communicators and revise our interpretations as listeners. As language models (LMs) are increasingly employed as dialogue interfaces and writing aids, handling ambiguous language is critical to their success. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence, and collect AmbiEnt, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity. We design a suite of tests based on AmbiEnt, presenting the first evaluation of pretrained LMs to recognize ambiguity and disentangle possible meanings. We find that the task remains extremely challenging, including for GPT-4, whose generated disambiguations are considered correct only 32% of the time in human evaluation, compared to 90% for disambiguations in our dataset. Finally, to illustrate the value of ambiguity-sensitive tools, we show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity. We encourage the field to rediscover the importance of ambiguity for NLP.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

30 of 53 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  10. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  11. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  12. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
18 more
  1. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  2. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  3. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  4. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  5. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  6. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  7. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 66
  8. 2016Ordinal Common-sense Inferencecited by 35
  9. 2019Uncertain Natural Language Inferencecited by 35
  10. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  11. 2022Investigating Reasons for Disagreement in Natural Language Inferencecited by 29
  12. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  13. 2022Task Ambiguity in Humans and Language Modelscited by 27
  14. 2021Embracing Ambiguity: Shifting the Training Target of NLI Modelscited by 22
  15. 2021Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Languagecited by 15
  16. 2022Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQAcited by 13
  17. 2023AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreferencecited by 12
  18. 2022Testing Pre-trained Language Models' Understanding of Distributivity via Causal Mediation Analysiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 33; 2025: 40; 2026: 28.

124 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  3. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  4. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  5. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  6. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  7. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
  8. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  9. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  10. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  11. 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 55
  12. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
38 more
  1. 2023SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tablescited by 51
  2. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  3. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  4. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  5. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 40
  6. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  7. 2023Can Large Language Models Capture Dissenting Human Voices?cited by 33
  8. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  9. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 33
  10. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33
  11. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  12. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 32
  13. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  14. 2023CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agentscited by 28
  15. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
  16. 2023Are Large Language Models Temporally Grounded?cited by 25
  17. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  18. 2025Towards Automating Data Access Permissions in AI Agentscited by 20
  19. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  20. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  21. 2023Zero and Few-shot Semantic Parsing with Ambiguous Inputscited by 16
  22. 2023Ecologically Valid Explanations for Label Variation in NLIcited by 16
  23. 2024Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?cited by 15
  24. 2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
  25. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  26. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13
  27. 2023Self-Consistency of Large Language Models under Ambiguitycited by 12
  28. 2025Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsingcited by 12
  29. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12
  30. 2024Scope Ambiguities in Large Language Modelscited by 11
  31. 2024HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoningcited by 11
  32. 2023Language Models as a Service: Overview of a New Paradigm and its Challengescited by 10
  33. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
  34. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  35. 2024Incremental Comprehension of Garden-Path Sentences by Large Language Models: Semantic Interpretation, Syntactic Re-Analysis, and Attentioncited by 10
  36. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
  37. 2025Do Large Language Models Understand Word Senses?cited by 10
  38. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9

50 most cited of 124 All 124 (JSON)