# Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives

arXiv: https://arxiv.org/abs/2410.05558 · HTML: https://mlsanity.com/paper/2410.05558/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 81 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Large Language Models Can Learn Temporal Reasoning](https://mlsanity.com/paper/2401.06853/) (2024)
- [TRAM: Benchmarking Temporal Reasoning for Large Language Models](https://mlsanity.com/paper/2310.00835/) (2023)
- [TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models](https://mlsanity.com/paper/2311.17667/) (2023)
- [Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems](https://mlsanity.com/paper/2404.14963/) (2024)
- [ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?](https://mlsanity.com/paper/2311.16989/) (2023)
- [Ethical ChatGPT: Concerns, Challenges, and Commandments](https://mlsanity.com/paper/2305.10646/) (2023)
- [Are Human-generated Demonstrations Necessary for In-context Learning?](https://mlsanity.com/paper/2309.14681/) (2023)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [The Poison of Alignment](https://mlsanity.com/paper/2308.13449/) (2023)
- [reStructured Pre-training](https://mlsanity.com/paper/2206.11147/) (2022)

## Built on by (most cited first, 5 of 5)

- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models](https://mlsanity.com/paper/2502.11425/) (2025)
- [Temporal reasoning for timeline summarisation in social media](https://mlsanity.com/paper/2501.00152/) (2024)
- [RAG-Enhanced Large Language Models for Dynamic Content Expiration Prediction in Web Search](https://mlsanity.com/paper/2605.13052/) (2026)
- [Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correction](https://mlsanity.com/paper/2509.12476/) (2025)

## Abstract (arXiv)

Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning -- temporal graph generation, to unveil LLMs' inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (<10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4. Our code is available at https://github.com/launchnlp/NoT.