The Imperfective Paradox Is Not Necessarily in Large Language Models: A Benchmark Failure Before a Model Failure
Not on the map. Why?
Abstract (arXiv)
The imperfective paradox provides a useful test of compositional semantic analysis. Recent work constructs an NLI benchmark and reports that models frequently infer completed telic events from progressive descriptions, attributing this behavior to a Teleological Bias. It further argues that prompting interventions cause a Calibration Crisis. We reexamine the benchmark and conclusions and show that it is substantially affected by conceptual and evaluation mis-specifications. We identify three conceptual mis-specifications. In particular, Aspectual Reduction affects the benchmark construction, analysis, experiments, and conclusions. Under a strict NLI standard, 76% of Group A instances do not explicitly rule out culmination. In our native-speaker annotation, 38% of Group A examples and 29% of the Group C examples were judged to permit an alternative interpretation. To control these issues and lexical variation, we construct Lexically Matched Minimal Pairs. At the evaluation level, we formulate event-semantic NLI as a Multi-step Reasoning Problem and assess both intermediate semantic decisions and final predictions. Our results show that models often do not affirm culmination but nevertheless accept the corresponding simple-past hypothesis, a pattern we characterize as Sufficiency Bias. We further show that prompting interventions produce a Decision Shift among labels without reliably improving the underlying semantic understanding and reasoning. Intermediate and oracle-guided analyses identify two additional failure modes: errors in compositional aspectual classification and Surface-form Attraction toward surface-associated answers. Our experiments on Qwen-7B with suitable prompts, GPT-5.4, and Qwen-72B provide initial evidence for the context sensitivity of aspectual classification and suggest that these models can achieve performance comparable to that of human annotators.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 25 Aug 2026.
Builds on
13 of 27 references.
References · 13
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
- 2018Annotation Artifacts in Natural Language Inference Datacited by 656
- 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
- 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
- 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
- 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
- 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
- 2025CoRE: Condition-based Reasoning for Identifying Outcome Variance in Complex Eventscited by 1
1 more
- 2026The Imperfective Paradox in Large Language Modelscited by 1
Built on by
Not cited yet.