mlsanityData to 7 Oct 2026

TRAM: Benchmarking Temporal Reasoning for Large Language Models

arXiv 2310.00835 ↗First posted 2 Oct 2023cs.CLcited by 39

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Reasoning about time is essential for understanding the nuances of events described in natural language. Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies. In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the TeR capabilities of large language models (LLMs). We evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios, and establish baselines with BERT-based and domain-specific models. Our findings indicate that the best-performing model lags significantly behind human performance. It is our aspiration that TRAM will spur further progress in enhancing the TeR capabilities of LLMs.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 30 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

25 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models15 Jun

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2023A Survey of Large Language Modelscited by 2,537
12 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  5. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  6. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  7. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  8. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  9. 2023Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understandingcited by 40
  10. 2022Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsiscited by 14
  11. 2021Empirical Quantitative Analysis of COVID-19 Forecasting Modelscited by 8
  12. 2022Enhancing Transformer Efficiency for Multivariate Time Series Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 12; 2026: 10.

39 papers.

Sort:

By area:

  1. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  2. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  3. 2023TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Modelscited by 39
  4. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  5. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31
  6. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  7. 2024Timo: Towards Better Temporal Reasoning for Language Modelscited by 21
  8. 2024Chain-of-History Reasoning for Temporal Knowledge Graph Forecastingcited by 17
  9. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  10. 2026TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Taskscited by 15
  11. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Modelscited by 13
  12. 2024CausalEval: Towards Better Causal Reasoning in Language Modelscited by 13
27 more
  1. 2025When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inferencecited by 12
  2. 2024Toward Conversational Agents with Context and Time Sensitive Long-term Memorycited by 8
  3. 2025Evaluating Long-Term Memory for Long-Context Question Answeringcited by 8
  4. 2024Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Modelscited by 7
  5. 2024A Comprehensive Evaluation of Large Language Models on Temporal Event Forecastingcited by 7
  6. 2024Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narrativescited by 5
  7. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  8. 2024LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Modelscited by 4
  9. 2025LLM-Symbolic Integration for Robust Temporal Tabular Reasoningcited by 4
  10. 2024Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Modelscited by 3
  11. 2025TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit Referencescited by 3
  12. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  13. 2024RASTeR: Robust, Agentic, and Structured Temporal Reasoningcited by 2
  14. 2025SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarioscited by 2
  15. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  16. 2026Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffingcited by 2
  17. 2025Rule Synergy Analysis using LLMs: State of the Art and Implicationscited by 1
  18. 2025Do Large Language Models (LLMs) Understand Chronology?cited by 1
  19. 2025NeSTR: A Neuro-Symbolic Abductive Framework for Temporal Reasoning in Large Language Modelscited by 1
  20. 2026Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Taskcited by 0
  21. 2026Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoningcited by 0
  22. 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
  23. 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudescited by 0
  24. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  25. 2026WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveformscited by 0
  26. 2026ChronoState: Hidden Elapsed-Time Conditioning for Temporal-State Action Selection in Frozen-Backbone Language Modelscited by 0
  27. 2026AI Historian: Helping historians organize and verify person-centred temporal clues from dispersed historical narrativescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios
  2. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time
  3. 2025ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events
  4. 2023MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models
  5. 2024Large Language Models Can Learn Temporal Reasoning
  6. 2025PRIMETIME : Limits of LLMs in Temporal Primitives
  7. 2023An Overview Of Temporal Commonsense Reasoning and Acquisition
  8. 2026What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?