mlsanityData to 14 Oct 2026

Scope Ambiguities in Large Language Models

arXiv 2404.04332 ↗First posted 5 Apr 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguities. These ambiguities offer rich insights into the interaction between semantic structure and world knowledge in language processing. Despite this, there has been little research into how modern large language models treat them. In this paper, we investigate how different versions of certain autoregressive language models -- GPT-2, GPT-3/3.5, Llama 2 and GPT-4 -- treat scope ambiguous sentences, and compare this with human judgments. We introduce novel datasets that contain a joint total of almost 1,000 unique scope-ambiguous sentences, containing interactions between a range of semantic operators, and annotated for human judgments. Using these datasets, we find evidence that several models (i) are sensitive to the meaning ambiguity in these sentences, in a way that patterns well with human judgments, and (ii) can successfully identify human-preferred readings at a high level of accuracy (over 90% in some cases).

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

29 of 46 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  6. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  7. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  8. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  9. 2022A Review on Language Models as Knowledge Basescited by 157
  10. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  11. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  12. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
17 more
  1. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 110
  2. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  3. 2020Syntactic Structure from Deep Learningcited by 99
  4. 2019Probing Natural Language Inference Models through Semantic Fragmentscited by 64
  5. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  6. 2019Can neural networks understand monotonicity reasoning?cited by 45
  7. 2019Probing What Different NLP Tasks Teach Machines about Function Word Comprehensioncited by 44
  8. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  9. 2020Assessing Phrasal Representation and Composition in Transformerscited by 41
  10. 2023Consistency Analysis of ChatGPTcited by 37
  11. 2020Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?cited by 29
  12. 2019Still a Pain in the Neck: Evaluating Text Representations on Lexical Compositioncited by 27
  13. 2018Assessing Composition in Sentence Vector Representationscited by 26
  14. 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 23
  15. 2023Zero and Few-shot Semantic Parsing with Ambiguous Inputscited by 16
  16. 2022Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarkscited by 8
  17. 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  2. 2024Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answeringcited by 13
  3. 2025QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewritingcited by 1
  4. 2026Generative AI for Managerial Decision-Making under Ambiguity and Sycophancycited by 1
  5. 2025PMG: Progressive Motion Generation via Sparse Anchor Postures Curriculum Learningcited by 0
  6. 2025Ambiguity in LLMs is a concept missing problemcited by 0
  7. 2025Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Modelscited by 0
  8. 2025Pragmatics beyond humans: meaning, communication, and LLMscited by 0
  9. 2025SeBERTis: A Framework for Producing Classifiers of Security-Related Issue Reportscited by 0
  10. 2026No One Size Fits All: QueryBandits for Hallucination Mitigationcited by 0
  11. 2026Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attributioncited by 0