# Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering

arXiv: https://arxiv.org/abs/2411.12395 · HTML: https://mlsanity.com/paper/2411.12395/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance](https://mlsanity.com/paper/2401.03729/) (2024)
- [Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?](https://mlsanity.com/paper/2308.01284/) (2023)
- [Task Ambiguity in Humans and Language Models](https://mlsanity.com/paper/2212.10711/) (2022)
- [Scope Ambiguities in Large Language Models](https://mlsanity.com/paper/2404.04332/) (2024)

## Built on by (most cited first, 13 of 13)

- [LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions](https://mlsanity.com/paper/2509.18970/) (2025)
- [Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering](https://mlsanity.com/paper/2502.13069/) (2025)
- [TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding](https://mlsanity.com/paper/2502.19400/) (2025)
- [Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2508.05687/) (2025)
- [VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction](https://mlsanity.com/paper/2602.13294/) (2026)
- [Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity](https://mlsanity.com/paper/2507.23121/) (2025)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [CascadeMind at SemEval-2026 Task 4: A Hybrid Neuro-Symbolic Cascade for Narrative Similarity](https://mlsanity.com/paper/2601.19931/) (2026)
- [Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation](https://mlsanity.com/paper/2603.01375/) (2026)
- [One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies](https://mlsanity.com/paper/2607.21143/) (2026)
- [Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective](https://mlsanity.com/paper/2506.16288/) (2025)
- [Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models](https://mlsanity.com/paper/2507.03726/) (2025)
- [CHAIRO: Contextual Hierarchical Analogical Induction and Reasoning Optimization for LLMs](https://mlsanity.com/paper/2604.10502/) (2026)

## Abstract (arXiv)

Ambiguity in natural language poses significant challenges to Large Language Models (LLMs) used for open-domain question answering. LLMs often struggle with the inherent uncertainties of human communication, leading to misinterpretations, miscommunications, hallucinations, and biased responses. This significantly weakens their ability to be used for tasks like fact-checking, question answering, feature extraction, and sentiment analysis. Using open-domain question answering as a test case, we compare off-the-shelf and few-shot LLM performance, focusing on measuring the impact of explicit disambiguation strategies. We demonstrate how simple, training-free, token-level disambiguation methods may be effectively used to improve LLM performance for ambiguous question answering tasks. We empirically show our findings and discuss best practices and broader impacts regarding ambiguity in LLMs.