# MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models

arXiv: https://arxiv.org/abs/2310.05157 · HTML: https://mlsanity.com/paper/2310.05157/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [Context-faithful Prompting for Large Language Models](https://mlsanity.com/paper/2303.11315/) (2023)
- [Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2306.08952/) (2023)
- [Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level](https://mlsanity.com/paper/2105.06020/) (2021)
- [Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios](https://mlsanity.com/paper/2305.16572/) (2023)
- [Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder](https://mlsanity.com/paper/2209.07692/) (2022)
- [S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering](https://mlsanity.com/paper/2305.11725/) (2023)
- [Multi-View Graph Representation Learning for Answering Hybrid Numerical Reasoning Question](https://mlsanity.com/paper/2305.03458/) (2023)
- [Learning to Imagine: Visually-Augmented Natural Language Generation](https://mlsanity.com/paper/2305.16944/) (2023)

## Built on by (most cited first, 15 of 15)

- [EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification](https://mlsanity.com/paper/2310.09754/) (2023)
- [AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2507.21836/) (2025)
- [Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?](https://mlsanity.com/paper/2406.09072/) (2024)
- [REMem: Reasoning with Episodic Memory in Language Agent](https://mlsanity.com/paper/2602.13530/) (2026)
- [TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions](https://mlsanity.com/paper/2502.21024/) (2025)
- [RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge](https://mlsanity.com/paper/2510.13590/) (2025)
- [LLMs as Repositories of Factual Knowledge: Limitations and Solutions](https://mlsanity.com/paper/2501.12774/) (2025)
- [Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models](https://mlsanity.com/paper/2504.05258/) (2025)
- [Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models](https://mlsanity.com/paper/2409.00617/) (2024)
- [MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models](https://mlsanity.com/paper/2510.19457/) (2025)
- [Relational Prompt-based Pre-trained Language Models for Social Event Detection](https://mlsanity.com/paper/2404.08263/) (2024)
- [Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges](https://mlsanity.com/paper/2505.11618/) (2025)
- [Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience](https://mlsanity.com/paper/2505.05515/) (2025)
- [Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Models](https://mlsanity.com/paper/2508.02045/) (2025)
- [Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity](https://mlsanity.com/paper/2512.00552/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time understanding and reasoning. However, research on the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests current mainstream LLMs with different parameter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall behind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show a significant vulnerability to temporal biases and depend heavily on the temporal information provided in questions. Furthermore, this paper undertakes a preliminary investigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches serve as valuable baselines or references for future research endeavors.