# How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs

arXiv: https://arxiv.org/abs/2507.14307 · HTML: https://mlsanity.com/paper/2507.14307/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [CogBench: a large language model walks into a psychology lab](https://mlsanity.com/paper/2402.18225/) (2024)
- [PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://mlsanity.com/paper/2401.07078/) (2024)
- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [The Dark Side of ChatGPT: Legal and Ethical Challenges from Stochastic Parrots and Hallucination](https://mlsanity.com/paper/2304.14347/) (2023)
- [Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts](https://mlsanity.com/paper/2309.17415/) (2023)
- [LLMs Are Prone to Fallacies in Causal Inference](https://mlsanity.com/paper/2406.12158/) (2024)
- [Do Language Models Exhibit Human-like Structural Priming Effects?](https://mlsanity.com/paper/2406.04847/) (2024)
- [Paraphrase Types Elicit Prompt Engineering Capabilities](https://mlsanity.com/paper/2406.19898/) (2024)
- [A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles](https://mlsanity.com/paper/2410.16139/) (2024)
- [CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans](https://mlsanity.com/paper/2406.15823/) (2024)
- [Using Artificial Populations to Study Psychological Phenomena in Neural Models](https://mlsanity.com/paper/2308.08032/) (2023)
- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [Large Language Model Recall Uncertainty is Modulated by the Fan Effect](https://mlsanity.com/paper/2407.06349/) (2024)
- [Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs](https://mlsanity.com/paper/2504.02789/) (2025)
- [Language Models Don't Learn the Physical Manifestation of Language](https://mlsanity.com/paper/2402.11349/) (2024)

## Built on by (most cited first, 3 of 3)

- [Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey](https://mlsanity.com/paper/2602.15851/) (2026)
- [Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?](https://mlsanity.com/paper/2512.07777/) (2025)
- [Taming the Centaur(s) with LAPITHS: a framework for a theoretically grounded interpretation of AI performances](https://mlsanity.com/paper/2604.27927/) (2026)

## Abstract (arXiv)

Large language models (LLMs) exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these behaviors reflect human-like cognition versus advanced pattern recognition remains an open question. In this study, we investigate how LLMs process the temporal meaning of linguistic aspect in narratives that were previously used in human studies. Using an Expert-in-the-Loop probing pipeline, we conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. Our findings show that LLMs over-rely on prototypicality, produce inconsistent aspectual judgments, and struggle with causal reasoning derived from aspect, raising concerns about their ability to fully comprehend narratives. These results suggest that LLMs process aspect fundamentally differently from humans and lack robust narrative understanding. Beyond these empirical findings, we develop a standardized experimental framework for the reliable assessment of LLMs' cognitive and linguistic capabilities.