Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility
Not on the map. Why?
Abstract (arXiv)
We present a hierarchy of natural language understanding abilities and argue for the importance of moving beyond assessments of understanding at the lexical and sentence levels to the discourse level. We propose the task of anaphora accessibility as a diagnostic for assessing discourse understanding, and to this end, present an evaluation dataset inspired by theoretical research in dynamic semantics. We evaluate human and LLM performance on our dataset and find that LLMs and humans align on some tasks and diverge on others. Such divergence can be explained by LLMs' reliance on specific lexical items during language comprehension, in contrast to human sensitivity to structural abstractions.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 10 of its 31 references that are in the map. First posted on arXiv 19 Feb 2025.
Builds on
10 of 31 references.
References · 10
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
- 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
- 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
- 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 77
- 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 23
- 2024TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Taskscited by 14
Built on by
Citing papers per year: 2025: 1.
1 paper.
- 2025AI-Assisted Authoring for Transparent, Data-Driven Documents27 Oct 2025