# Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning

arXiv: https://arxiv.org/abs/2311.09821 · HTML: https://mlsanity.com/paper/2311.09821/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 7 of 7)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER](https://mlsanity.com/paper/2108.13655/) (2021)
- [QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs](https://mlsanity.com/paper/2205.12665/) (2022)

## Built on by (most cited first, 18 of 18)

- [Large Language Models Can Learn Temporal Reasoning](https://mlsanity.com/paper/2401.06853/) (2024)
- [REMem: Reasoning with Episodic Memory in Language Agent](https://mlsanity.com/paper/2602.13530/) (2026)
- [Timo: Towards Better Temporal Reasoning for Language Models](https://mlsanity.com/paper/2406.14192/) (2024)
- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [Enhancing Temporal Understanding in LLMs for Semi-structured Tables](https://mlsanity.com/paper/2407.16030/) (2024)
- [DyG-RAG: Dynamic Graph Retrieval-Augmented Generation with Event-Centric Reasoning](https://mlsanity.com/paper/2507.13396/) (2025)
- [Around the World in 24 Hours: Probing LLM Knowledge of Time and Place](https://mlsanity.com/paper/2506.03984/) (2025)
- [Date Fragments: A Hidden Bottleneck of Tokenization for Temporal Reasoning](https://mlsanity.com/paper/2505.16088/) (2025)
- [Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models](https://mlsanity.com/paper/2402.11997/) (2024)
- [BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence](https://mlsanity.com/paper/2603.07931/) (2026)
- [DateLogicQA: Benchmarking Temporal Biases in Large Language Models](https://mlsanity.com/paper/2412.13377/) (2024)
- [Leveraging LLM For Synchronizing Information Across Multilingual Tables](https://mlsanity.com/paper/2504.02559/) (2025)
- [Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models](https://mlsanity.com/paper/2606.20959/) (2026)
- [Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.05939/) (2025)
- [Structured yet Bounded Temporal Understanding in Large Language Models](https://mlsanity.com/paper/2510.16685/) (2025)
- [TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models](https://mlsanity.com/paper/2601.14951/) (2026)
- [Executable Schema Contracts: From Automatic Ingestion to Multi-Source Retrieval](https://mlsanity.com/paper/2606.05415/) (2026)
- [ScalableRAG: High-Quality RAG at Zero Ingestion Cost](https://mlsanity.com/paper/2607.25135/) (2026)

## Abstract (arXiv)

Knowledge in the real world is being updated constantly. However, it is costly to frequently update large language models (LLMs). Therefore, it is crucial for LLMs to understand the concept of temporal knowledge. However, prior works on temporal question answering (TQA) did not emphasize multi-answer and multi-hop types of temporal reasoning. In this paper, we propose a complex temporal question-answering dataset Complex-TR that focuses on multi-answer and multi-hop temporal reasoning. Besides, we also propose a novel data augmentation strategy to improve the complex temporal reasoning capability and robustness of LLMs. We conducted experiments on multiple temporal QA datasets. Experimental results show that our method is able to improve LLMs' performance on temporal QA benchmarks by significant margins. Our code and data are released at: https://github.com/nusnlp/complex-tr.