mlsanityData to 7 Oct 2026

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

arXiv 2406.09170 ↗First posted 13 Jun 2024cs.CLcited by 83

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 72 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

55 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models19 Feb
  2. 2024Large Language Models Can Learn Temporal Reasoning12 Jan
  3. 2023TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models29 Nov
  4. 2023Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoning23 Oct
  5. 2023TRAM: Benchmarking Temporal Reasoning for Large Language Models2 Oct
  6. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models15 Jun

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2023A Survey of Large Language Modelscited by 2,537
37 more
  1. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  6. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  7. 2023Language Models Represent Space and Timecited by 306
  8. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  9. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  10. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  11. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  12. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  13. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  14. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  15. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  16. 2021SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networkscited by 108
  17. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  18. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  19. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  20. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  21. 2021Complex Temporal Question Answering on Knowledge Graphscited by 69
  22. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  23. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  24. 2019Joint Reasoning for Temporal and Causal Relationscited by 60
  25. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  26. 2022GraphWorld: Fake Graphs Bring Real Insights for GNNscited by 56
  27. 2023Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environmentscited by 52
  28. 2023Do Large Language Models Know about Facts?cited by 49
  29. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  30. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  31. 2020Grale: Designing Networks for Graph Learningcited by 28
  32. 2023TempTabQA: Temporal Question Answering for Semi-Structured Tablescited by 27
  33. 2020Pathfinder Discovery Networks for Neural Message Passingcited by 16
  34. 2022Synthetic Graph Generation to Benchmark Graph Learningcited by 16
  35. 2023UGSL: A Unified Framework for Benchmarking Graph Structure Learningcited by 9
  36. 2021SYGMA: System for Generalizable Modular Question Answering OverKnowledge Basescited by 8
  37. 2023Examining the Effects of Degree Distribution and Homophily in Graph Learning Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 44; 2026: 29.

83 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2025BIG-Bench Extra Hardcited by 123
  4. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  5. 2026REMem: Reasoning with Episodic Memory in Language Agentcited by 20
  6. 2024UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorizationcited by 17
  7. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoningcited by 17
  8. 2025ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Eventscited by 17
  9. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologiescited by 17
  10. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!cited by 15
  11. 2026TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Taskscited by 15
  12. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 13
38 more
  1. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  2. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  3. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Timecited by 12
  4. 2025It's High Time: A Survey of Temporal Question Answeringcited by 12
  5. 2025When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inferencecited by 12
  6. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  7. 2025Inferring Events from Time Series using Language Modelscited by 10
  8. 2025A Study into Investigating Temporal Robustness of LLMscited by 10
  9. 2025Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Modelscited by 9
  10. 2026KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companionscited by 9
  11. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  12. 2025TCP: a Benchmark for Temporal Constraint-Based Planningcited by 8
  13. 2024ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domainscited by 7
  14. 2025Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perceptioncited by 7
  15. 2025Date Fragments: A Hidden Bottleneck of Tokenization for Temporal Reasoningcited by 6
  16. 2025Around the World in 24 Hours: Probing LLM Knowledge of Time and Placecited by 6
  17. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 6
  18. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  19. 2025ExAnte: A Benchmark for Ex-Ante Inference in Large Language Modelscited by 5
  20. 2025Visual Large Language Models Exhibit Human-Level Cognitive Flexibility in the Wisconsin Card Sorting Testcited by 5
  21. 2025LLMAP: LLM-Assisted Multi-Objective Route Planning with User Preferencescited by 5
  22. 2024LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Modelscited by 4
  23. 2025MastermindEval: A Simple But Scalable Reasoning Benchmarkcited by 4
  24. 2025Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMscited by 4
  25. 2025Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applicationscited by 4
  26. 2025Exploring the Potential of Large Language Models as Predictors in Dynamic Text-Attributed Graphscited by 3
  27. 2026When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysiscited by 3
  28. 2024DateLogicQA: Benchmarking Temporal Biases in Large Language Modelscited by 2
  29. 2025On the Temporal Question-Answering Capabilities of Large Language Models Over Anonymized Datacited by 2
  30. 2025TiMoE: Time-Aware Mixture of Language Expertscited by 2
  31. 2025Towards Interpretable and Trustworthy Time Series Reasoning: A BlueSky Visioncited by 2
  32. 2025When Facts Change: Probing LLMs on Evolving Knowledge with evolveQAcited by 2
  33. 2025Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generationcited by 2
  34. 2025Critical Confabulation: Can LLMs Hallucinate for Social Good?cited by 2
  35. 2026Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoningcited by 2
  36. 2026Reasoning Gets Harder for LLMs Inside A Dialoguecited by 2
  37. 2026STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMscited by 2
  38. 2026Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Modelscited by 2

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios
  2. 2023MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models
  3. 2025TransientTables: Evaluating LLMs' Reasoning on Temporally Evolving Semi-structured Tables
  4. 2025TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References
  5. 2024Enhancing Temporal Understanding in LLMs for Semi-structured Tables
  6. 2023Unlocking Temporal Question Answering for Large Language Models with Tailor-Made Reasoning Logic
  7. 2025If I Could Turn Back Time: Temporal Reframing as a Historical Reasoning Task for LLMs
  8. 2025An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint