mlsanityData to 7 Oct 2026

NOPE: A Corpus of Naturally-Occurring Presuppositions in English

arXiv 2109.06987 ↗First posted 14 Sep 2021cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid. These inferences include presuppositions, a phenomenon by which a listener learns about new information through reasoning about what a speaker takes as given. Presuppositions require complex understanding of the lexical and syntactic properties that trigger them as well as the broader conversational context. In this work, we introduce the Naturally-Occurring Presuppositions in English (NOPE) Corpus to investigate the context-sensitivity of 10 different types of presupposition triggers and to evaluate machine learning models' ability to predict human inferences. We find that most of the triggers we investigate exhibit moderate variability. We further find that transformer-based models draw correct inferences in simple cases involving presuppositions, but they fail to capture the minority of exceptional cases in which human judgments reveal complex interactions between context and triggers.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 17 of its 71 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

17 of 71 references.

References · 17

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  10. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  11. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  12. 2017Advances in Pre-Training Distributed Word Representationscited by 207
5 more
  1. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  2. 2021Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answeringcited by 41
  3. 2019Uncertain Natural Language Inferencecited by 35
  4. 2019Harnessing the linguistic signal to predict scalar inferencescited by 17
  5. 2018Lexicosyntactic Inference in Neural Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 4; 2025: 4; 2026: 4.

23 papers.

Sort:

By area:

  1. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  2. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  3. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  4. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  5. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  6. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 17
  7. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  8. 2023Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answeringcited by 13
  9. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  10. 2025Linguistic Generalizations are not Rules: Impacts on Evaluation of LMscited by 10
  11. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  12. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blindscited by 7
11 more
  1. 2023Evaluating Paraphrastic Robustness in Textual Entailment Modelscited by 7
  2. 2024Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctionscited by 5
  3. 2023Towards Understanding What Code Language Models Learnedcited by 3
  4. 2025Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presuppositioncited by 3
  5. 2026Do Language Models Know Theo Has a Wife? Investigating the Proviso Problemcited by 1
  6. 2026It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Beliefcited by 1
  7. 2022Polish Natural Language Inference and Factivity -- an Expert-based Dataset and Benchmarkscited by 0
  8. 2023PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environmentscited by 0
  9. 2024LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognitioncited by 0
  10. 2026DRInQ: Evaluating Conversational Implicature with Controlled Context Variationcited by 0
  11. 2026Pragmatic Attack Surface: Vulnerabilities of Implicit Context in Large Language Modelscited by 0