# A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting

arXiv: https://arxiv.org/abs/2407.11638 · HTML: https://mlsanity.com/paper/2407.11638/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Retrieval-augmented generation / [Time series and agricultural RAG](https://mlsanity.com/rag/time-series-and-agricultural-rag/)

## Builds on (most cited first, 30 of 35)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs](https://mlsanity.com/paper/1705.05742/) (2017)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completion](https://mlsanity.com/paper/1811.04441/) (2018)
- ["Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding](https://mlsanity.com/paper/1909.03065/) (2019)
- [Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning](https://mlsanity.com/paper/2104.10353/) (2021)
- [Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks](https://mlsanity.com/paper/2012.08492/) (2020)
- [A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal](https://mlsanity.com/paper/2005.10070/) (2020)
- [TimeTraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting](https://mlsanity.com/paper/2109.04101/) (2021)
- [TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions](https://mlsanity.com/paper/2005.00242/) (2020)
- [Temporal Knowledge Graph Completion: A Survey](https://mlsanity.com/paper/2201.08236/) (2022)
- [Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning](https://mlsanity.com/paper/2305.16646/) (2023)
- [ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data](https://mlsanity.com/paper/2005.00792/) (2020)
- [The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction](https://mlsanity.com/paper/2104.06344/) (2021)
- [TRAM: Benchmarking Temporal Reasoning for Large Language Models](https://mlsanity.com/paper/2310.00835/) (2023)
- [Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.00327/) (2021)
- [Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning](https://mlsanity.com/paper/2305.10613/) (2023)
- [Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2306.08952/) (2023)

## Built on by (most cited first, 8 of 8)

- [OpenEP: Open-Ended Future Event Prediction](https://mlsanity.com/paper/2408.06578/) (2024)
- [OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data](https://mlsanity.com/paper/2510.15096/) (2025)
- [Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting](https://mlsanity.com/paper/2505.09852/) (2025)
- [AILS-NTUA at SemEval-2026 Task 12: Graph-Based Retrieval and Reflective Prompting for Abductive Event Reasoning](https://mlsanity.com/paper/2603.04319/) (2026)
- [A Multi-Expert Structural-Semantic Hybrid Framework for Unveiling Historical Patterns in Temporal Knowledge Graphs](https://mlsanity.com/paper/2506.14235/) (2025)
- [ForecastAgentSearch: Towards a Multi-Expert Agent Search System for Geopolitical Event Forecasting](https://mlsanity.com/paper/2606.31665/) (2026)
- [A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts](https://mlsanity.com/paper/2606.27881/) (2026)
- [Can LLMs Access their World Knowledge for Event Prediction?](https://mlsanity.com/paper/2610.19021/) (2026)

## Abstract (arXiv)

Recently, Large Language Models (LLMs) have demonstrated great potential in various data mining tasks, such as knowledge question answering, mathematical reasoning, and commonsense reasoning. However, the reasoning capability of LLMs on temporal event forecasting has been under-explored. To systematically investigate their abilities in temporal event forecasting, we conduct a comprehensive evaluation of LLM-based methods for temporal event forecasting. Due to the lack of a high-quality dataset that involves both graph and textual data, we first construct a benchmark dataset, named MidEast-TE-mini. Based on this dataset, we design a series of baseline methods, characterized by various input formats and retrieval augmented generation (RAG) modules. From extensive experiments, we find that directly integrating raw texts into the input of LLMs does not enhance zero-shot extrapolation performance. In contrast, fine-tuning LLMs with raw texts can significantly improve performance. Additionally, LLMs enhanced with retrieval modules can effectively capture temporal relational patterns hidden in historical events. However, issues such as popularity bias and the long-tail problem persist in LLMs, particularly in the retrieval-augmented generation (RAG) method. These findings not only deepen our understanding of LLM-based event forecasting methods but also highlight several promising research directions. We consider that this comprehensive evaluation, along with the identified research opportunities, will significantly contribute to future research on temporal event forecasting through LLMs.