# Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

arXiv: https://arxiv.org/abs/2610.17402 · HTML: https://mlsanity.com/paper/2610.17402/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 24 of 24)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- ["Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding](https://mlsanity.com/paper/1909.03065/) (2019)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [Joint Reasoning for Temporal and Causal Relations](https://mlsanity.com/paper/1906.04941/) (2019)
- [Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction](https://mlsanity.com/paper/1909.05360/) (2019)
- [Zero-shot Temporal Relation Extraction with ChatGPT](https://mlsanity.com/paper/2304.05454/) (2023)
- [Temporal Common Sense Acquisition with Minimal Supervision](https://mlsanity.com/paper/2005.04304/) (2020)
- [Are Large Language Models Temporally Grounded?](https://mlsanity.com/paper/2311.08398/) (2023)
- [A Structured Learning Approach to Temporal Relation Extraction](https://mlsanity.com/paper/1906.04943/) (2019)
- [Deep Structured Neural Network for Event Temporal Relation Extraction](https://mlsanity.com/paper/1909.10094/) (2019)
- [Fine-Grained Temporal Relation Extraction](https://mlsanity.com/paper/1902.01390/) (2019)
- [ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events](https://mlsanity.com/paper/2501.03040/) (2025)
- [More than Classification: A Unified Framework for Event Temporal Relation Extraction](https://mlsanity.com/paper/2305.17607/) (2023)
- [Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning](https://mlsanity.com/paper/2305.14970/) (2023)
- [Conditional Generation of Temporally-ordered Event Sequences](https://mlsanity.com/paper/2012.15786/) (2020)
- [A Survey on Temporal Reasoning for Temporal Information Extraction from Text (Extended Abstract)](https://mlsanity.com/paper/2005.06527/) (2020)
- [LexTime: A Benchmark for Temporal Ordering of Legal Events](https://mlsanity.com/paper/2506.04041/) (2025)
- [TIMELINE: Exhaustive Annotation of Temporal Relations Supporting the Automatic Ordering of Events in News Articles](https://mlsanity.com/paper/2310.17802/) (2023)
- [Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency](https://mlsanity.com/paper/2406.11486/) (2024)
- [Temporal Event Knowledge Acquisition via Identifying Narratives](https://mlsanity.com/paper/1805.10956/) (2018)
- [Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models](https://mlsanity.com/paper/2502.11425/) (2025)
- [Beyond Pairwise: Global Zero-shot Temporal Graph Generation](https://mlsanity.com/paper/2502.11114/) (2025)
- [EventFull: Complete and Consistent Event Relation Annotation](https://mlsanity.com/paper/2412.12733/) (2024)
- [QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi](https://mlsanity.com/paper/2605.18380/) (2026)

## Abstract (arXiv)

Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.