# TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models

arXiv: https://arxiv.org/abs/2311.17667 · HTML: https://mlsanity.com/paper/2311.17667/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 59 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents](https://mlsanity.com/paper/2311.11797/) (2023)
- [TRAM: Benchmarking Temporal Reasoning for Large Language Models](https://mlsanity.com/paper/2310.00835/) (2023)
- [Are Large Language Models Temporally Grounded?](https://mlsanity.com/paper/2311.08398/) (2023)
- [GLoRE: Evaluating Logical Reasoning of Large Language Models](https://mlsanity.com/paper/2310.09107/) (2023)

## Built on by (most cited first, 30 of 39)

- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2405.14170/) (2024)
- [Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?](https://mlsanity.com/paper/2406.09072/) (2024)
- [Time-R1: Towards Comprehensive Temporal Reasoning in LLMs](https://mlsanity.com/paper/2505.13508/) (2025)
- [TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks](https://mlsanity.com/paper/2602.13272/) (2026)
- [REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Tasks](https://mlsanity.com/paper/2502.18836/) (2025)
- [TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions](https://mlsanity.com/paper/2502.21024/) (2025)
- [Temporal Knowledge Graph Question Answering: A Survey](https://mlsanity.com/paper/2406.14191/) (2024)
- [Rethinking Emotion Annotations in the Era of Large Language Models](https://mlsanity.com/paper/2412.07906/) (2024)
- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models](https://mlsanity.com/paper/2505.19533/) (2025)
- [Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives](https://mlsanity.com/paper/2410.05558/) (2024)
- [ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning](https://mlsanity.com/paper/2505.12501/) (2025)
- [USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents](https://mlsanity.com/paper/2505.17572/) (2025)
- [LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Models](https://mlsanity.com/paper/2407.05434/) (2024)
- [Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions](https://mlsanity.com/paper/2508.13214/) (2025)
- [TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References](https://mlsanity.com/paper/2505.01325/) (2025)
- [PMOA-TTS: Introducing the PubMed Open Access Textual Times Series Corpus](https://mlsanity.com/paper/2505.20323/) (2025)
- [ALAS: Transactional and Dynamic Multi-Agent LLM Planning](https://mlsanity.com/paper/2511.03094/) (2025)
- [RASTeR: Robust, Agentic, and Structured Temporal Reasoning](https://mlsanity.com/paper/2406.19538/) (2024)
- [TiMoE: Time-Aware Mixture of Language Experts](https://mlsanity.com/paper/2508.08827/) (2025)
- [Critical Confabulation: Can LLMs Hallucinate for Social Good?](https://mlsanity.com/paper/2511.07722/) (2025)
- [SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning](https://mlsanity.com/paper/2512.16461/) (2025)
- [Causal Stories from Sensor Traces: Auditing Epistemic Overreach in LLM-Generated Personal Sensing Explanations](https://mlsanity.com/paper/2605.08590/) (2026)
- [Self-Exploring Language Models for Explainable Link Forecasting on Temporal Graphs via Reinforcement Learning](https://mlsanity.com/paper/2509.00975/) (2025)
- [NeSTR: A Neuro-Symbolic Abductive Framework for Temporal Reasoning in Large Language Models](https://mlsanity.com/paper/2512.07218/) (2025)
- [Three Generations of Healthcare IT: From the Digital Record to the Computable Care Process](https://mlsanity.com/paper/2608.08806/) (2026)
- [LTLZinc: a Benchmarking Framework for Continual Learning and Neuro-Symbolic Temporal Reasoning](https://mlsanity.com/paper/2507.17482/) (2025)
- [Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs](https://mlsanity.com/paper/2509.24640/) (2025)
- [CaMiT: A Time-Aware Car Model Dataset for Classification and Generation](https://mlsanity.com/paper/2510.17626/) (2025)

## Abstract (arXiv)

Grasping the concept of time is a fundamental facet of human cognition, indispensable for truly comprehending the intricacies of the world. Previous studies typically focus on specific aspects of time, lacking a comprehensive temporal reasoning benchmark. To address this, we propose TimeBench, a comprehensive hierarchical temporal reasoning benchmark that covers a broad spectrum of temporal reasoning phenomena. TimeBench provides a thorough evaluation for investigating the temporal reasoning capabilities of large language models. We conduct extensive experiments on GPT-4, LLaMA2, and other popular LLMs under various settings. Our experimental results indicate a significant performance gap between the state-of-the-art LLMs and humans, highlighting that there is still a considerable distance to cover in temporal reasoning. Besides, LLMs exhibit capability discrepancies across different reasoning categories. Furthermore, we thoroughly analyze the impact of multiple aspects on temporal reasoning and emphasize the associated challenges. We aspire for TimeBench to serve as a comprehensive benchmark, fostering research in temporal reasoning. Resources are available at: https://github.com/zchuz/TimeBench