mlsanityData to 14 Oct 2026

MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models

arXiv 2310.05157 ↗First posted 8 Oct 2023cs.CL cs.AIcited by 15

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time understanding and reasoning. However, research on the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests current mainstream LLMs with different parameter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall behind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show a significant vulnerability to temporal biases and depend heavily on the temporal information provided in questions. Furthermore, this paper undertakes a preliminary investigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches serve as valuable baselines or references for future research endeavors.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 27 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

24 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models15 Jun

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
11 more
  1. 2022RealTime QA: What's the Answer Right Now?cited by 244
  2. 2021A Dataset for Answering Time-Sensitive Questionscited by 154
  3. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  4. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  5. 2023Context-faithful Prompting for Large Language Modelscited by 80
  6. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  7. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarioscited by 27
  8. 2022Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decodercited by 17
  9. 2023S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answeringcited by 17
  10. 2023Multi-View Graph Representation Learning for Answering Hybrid Numerical Reasoning Questioncited by 10
  11. 2023Learning to Imagine: Visually-Augmented Natural Language Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 10; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023EX-FEVER: A Dataset for Multi-hop Explainable Fact Verificationcited by 36
  2. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 33
  3. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  4. 2026REMem: Reasoning with Episodic Memory in Language Agentcited by 23
  5. 2025TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questionscited by 15
  6. 2025RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledgecited by 13
  7. 2025LLMs as Repositories of Factual Knowledge: Limitations and Solutionscited by 10
  8. 2025Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Modelscited by 10
  9. 2024Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Modelscited by 9
  10. 2025MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Modelscited by 7
  11. 2024Relational Prompt-based Pre-trained Language Models for Social Event Detectioncited by 5
  12. 2025Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challengescited by 4
3 more
  1. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  2. 2025Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Modelscited by 0
  3. 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
  2. 2025TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios
  3. 2025A Study into Investigating Temporal Robustness of LLMs
  4. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time
  5. 2023TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models
  6. 2023Unlocking Temporal Question Answering for Large Language Models with Tailor-Made Reasoning Logic
  7. 2026Is More Context Always Better? Examining LLM Reasoning Capability for Time Interval Prediction
  8. 2025TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References