# PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimation

arXiv: https://arxiv.org/abs/2504.01509 · HTML: https://mlsanity.com/paper/2504.01509/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2407.11005/) (2024)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models](https://mlsanity.com/paper/2210.12023/) (2022)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [e-CARE: a New Dataset for Exploring Explainable Causal Reasoning](https://mlsanity.com/paper/2205.05849/) (2022)
- [AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework](https://mlsanity.com/paper/2403.12582/) (2024)
- [Distilling Script Knowledge from Large Language Models for Constrained Language Planning](https://mlsanity.com/paper/2305.05252/) (2023)
- [Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering](https://mlsanity.com/paper/2402.14320/) (2024)
- [Can Language Models Use Forecasting Strategies?](https://mlsanity.com/paper/2406.04446/) (2024)
- [OpenEP: Open-Ended Future Event Prediction](https://mlsanity.com/paper/2408.06578/) (2024)
- [Exploring Large Language Models for Climate Forecasting](https://mlsanity.com/paper/2411.13724/) (2024)
- [Reasoning and Tools for Human-Level Forecasting](https://mlsanity.com/paper/2408.12036/) (2024)
- [WeQA: A Benchmark for Retrieval Augmented Generation in Wind Energy Domain](https://mlsanity.com/paper/2408.11800/) (2024)
- [Language Models as Causal Effect Generators](https://mlsanity.com/paper/2411.08019/) (2024)
- [VERA: Validation and Enhancement for Retrieval Augmented systems](https://mlsanity.com/paper/2409.15364/) (2024)

## Built on by (most cited first, 8 of 8)

- [Pitfalls in Evaluating Language Model Forecasters](https://mlsanity.com/paper/2506.00723/) (2025)
- [AIA Forecaster: Technical Report](https://mlsanity.com/paper/2511.07678/) (2025)
- [Scientific reasoning does not reliably translate into scientific forecasting in frontier AI](https://mlsanity.com/paper/2605.22681/) (2026)
- [FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewards](https://mlsanity.com/paper/2604.26733/) (2026)
- [ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment](https://mlsanity.com/paper/2606.00644/) (2026)
- [Harnessing Pre-Resolution Signals for Future Prediction Agents](https://mlsanity.com/paper/2604.15719/) (2026)
- [Distilling Temporal Search and Reasoning: Evolving LLMs for Future Prediction via Harness-Assisted Efficient Data Synthesis](https://mlsanity.com/paper/2607.25554/) (2026)
- [CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference](https://mlsanity.com/paper/2609.36689/) (2026)

## Abstract (arXiv)

Predicting future events based on news on the Web stands as one of the ultimate aspirations of artificial intelligence. Recent advances in large language model (LLM)-based systems have shown remarkable potential in forecasting future events, thereby garnering significant interest in the research community. Currently, several benchmarks have been established to evaluate the forecasting capabilities by formalizing the event prediction as a retrieval-augmented generation (RAG)-and-reasoning task. In these benchmarks, each prediction question is answered with relevant retrieved news articles downloaded from the Web. However, because there is no consideration of whether the questions can be supported by valid or sufficient supporting rationales, some of the questions in these benchmarks may be inherently noninferable. To address this issue, we introduce a new benchmark, PROPHET, which comprises inferable forecasting questions paired with relevant news for retrieval. To ensure the inferability of the benchmark, we propose Causal Intervened Likelihood (CIL), a statistical measure that assesses inferability through causal inference. In constructing this benchmark, we first collected recent trend forecasting questions, and then filtered the data using CIL resulting in an inferable benchmark for future forecasting. Through extensive experiments, we first demonstrate the validity of CIL and in-depth investigations into future forecasting with the aid of CIL. Subsequently, we evaluate several representative prediction methods on PROPHET. The overall results draws valuable insights for task of future directions.