# TRAM: Benchmarking Temporal Reasoning for Large Language Models

arXiv: https://arxiv.org/abs/2310.00835 · HTML: https://mlsanity.com/paper/2310.00835/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- ["Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding](https://mlsanity.com/paper/1909.03065/) (2019)
- [Metacognitive Prompting Improves Understanding in Large Language Models](https://mlsanity.com/paper/2308.05342/) (2023)
- [Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding](https://mlsanity.com/paper/2304.05368/) (2023)
- [Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2306.08952/) (2023)
- [Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis](https://mlsanity.com/paper/2203.14469/) (2022)
- [Empirical Quantitative Analysis of COVID-19 Forecasting Models](https://mlsanity.com/paper/2110.00174/) (2021)
- [Enhancing Transformer Efficiency for Multivariate Time Series Classification](https://mlsanity.com/paper/2203.14472/) (2022)

## Built on by (most cited first, 30 of 40)

- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [MIRAI: Evaluating LLM Agents for Event Forecasting](https://mlsanity.com/paper/2407.01231/) (2024)
- [TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models](https://mlsanity.com/paper/2311.17667/) (2023)
- [Graph-enhanced Large Language Models in Asynchronous Plan Reasoning](https://mlsanity.com/paper/2402.02805/) (2024)
- [Robotouille: An Asynchronous Planning Benchmark for LLM Agents](https://mlsanity.com/paper/2502.05227/) (2025)
- [Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?](https://mlsanity.com/paper/2406.09072/) (2024)
- [Timo: Towards Better Temporal Reasoning for Language Models](https://mlsanity.com/paper/2406.14192/) (2024)
- [Chain-of-History Reasoning for Temporal Knowledge Graph Forecasting](https://mlsanity.com/paper/2402.14382/) (2024)
- [TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks](https://mlsanity.com/paper/2602.13272/) (2026)
- [Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models](https://mlsanity.com/paper/2312.17661/) (2023)
- [RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models](https://mlsanity.com/paper/2406.11020/) (2024)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference](https://mlsanity.com/paper/2509.01822/) (2025)
- [Toward Conversational Agents with Context and Time Sensitive Long-term Memory](https://mlsanity.com/paper/2406.00057/) (2024)
- [A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting](https://mlsanity.com/paper/2407.11638/) (2024)
- [Evaluating Long-Term Memory for Long-Context Question Answering](https://mlsanity.com/paper/2510.23730/) (2025)
- [Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models](https://mlsanity.com/paper/2406.10584/) (2024)
- [Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives](https://mlsanity.com/paper/2410.05558/) (2024)
- [USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents](https://mlsanity.com/paper/2505.17572/) (2025)
- [Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models](https://mlsanity.com/paper/2402.11997/) (2024)
- [LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Models](https://mlsanity.com/paper/2407.05434/) (2024)
- [LLM-Symbolic Integration for Robust Temporal Tabular Reasoning](https://mlsanity.com/paper/2506.05746/) (2025)
- [SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios](https://mlsanity.com/paper/2503.06218/) (2025)
- [TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References](https://mlsanity.com/paper/2505.01325/) (2025)
- [Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience](https://mlsanity.com/paper/2505.05515/) (2025)
- [RASTeR: Robust, Agentic, and Structured Temporal Reasoning](https://mlsanity.com/paper/2406.19538/) (2024)
- [SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning](https://mlsanity.com/paper/2512.16461/) (2025)
- [From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes](https://mlsanity.com/paper/2605.16303/) (2026)
- [Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing](https://mlsanity.com/paper/2605.25902/) (2026)
- [Rule Synergy Analysis using LLMs: State of the Art and Implications](https://mlsanity.com/paper/2508.19484/) (2025)

## Abstract (arXiv)

Reasoning about time is essential for understanding the nuances of events described in natural language. Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies. In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the TeR capabilities of large language models (LLMs). We evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios, and establish baselines with BERT-based and domain-specific models. Our findings indicate that the best-performing model lags significantly behind human performance. It is our aspiration that TRAM will spur further progress in enhancing the TeR capabilities of LLMs.