# Bench to the Future: A Pastcasting Benchmark for Forecasting Agents

arXiv: https://arxiv.org/abs/2506.21558 · HTML: https://mlsanity.com/paper/2506.21558/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 19 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 6 of 6)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Deep Research Bench: Evaluating AI Web Research Agents](https://mlsanity.com/paper/2506.06287/) (2025)
- [Reasoning and Tools for Human-Level Forecasting](https://mlsanity.com/paper/2408.12036/) (2024)

## Built on by (most cited first, 21 of 21)

- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](https://mlsanity.com/paper/2510.17638/) (2025)
- [Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts](https://mlsanity.com/paper/2507.19477/) (2025)
- [Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff](https://mlsanity.com/paper/2601.13717/) (2026)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [Automating Forecasting Question Generation and Resolution for AI Evaluation](https://mlsanity.com/paper/2601.22444/) (2026)
- [KellyBench: A Benchmark for Long-Horizon Sequential Decision Making](https://mlsanity.com/paper/2604.27865/) (2026)
- [Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting](https://mlsanity.com/paper/2605.27904/) (2026)
- [WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning](https://mlsanity.com/paper/2606.11816/) (2026)
- [ForecastBench-Sim: A Simulated-World Forecasting Benchmark](https://mlsanity.com/paper/2606.18686/) (2026)
- [Agentic Time Machine as an Infrastructure for Future-Event Forecasting](https://mlsanity.com/paper/2606.21013/) (2026)
- [Harnessing Pre-Resolution Signals for Future Prediction Agents](https://mlsanity.com/paper/2604.15719/) (2026)
- [Analytica: Soft Propositional Reasoning for Robust and Scalable LLM-Driven Analysis](https://mlsanity.com/paper/2604.23072/) (2026)
- [Evaluating Strategic Reasoning in Forecasting Agents](https://mlsanity.com/paper/2604.26106/) (2026)
- [Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Study](https://mlsanity.com/paper/2602.00758/) (2026)
- [Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most](https://mlsanity.com/paper/2605.22672/) (2026)
- [Global Merger-Arbitrage Forecasting with Language Models](https://mlsanity.com/paper/2607.09921/) (2026)
- [Diverse by Reasoning: Harnessing the Wisdom of LLM Crowds for Future Prediction](https://mlsanity.com/paper/2608.24001/) (2026)
- [Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents](https://mlsanity.com/paper/2609.28876/) (2026)
- [Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts](https://mlsanity.com/paper/2609.33257/) (2026)
- [CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference](https://mlsanity.com/paper/2609.36689/) (2026)

## Abstract (arXiv)

Forecasting is a challenging task that offers a clearly measurable way to study AI systems. Forecasting requires a large amount of research on the internet, and evaluations require time for events to happen, making the development of forecasting benchmarks challenging. To date, no forecasting benchmark provides a realistic, hermetic, and repeatable environment for LLM forecasters. We introduce Bench To the Future (BTF), a "pastcasting" benchmark with hundreds of high-quality questions for which the resolution is already known. Each question is accompanied by a large offline corpus of tens of thousands of relevant web pages, enabling a way to elicit realistic "forecasts" on past events from LLMs. Results suggest that our pastcasting environment can produce results comparable to those based on forecasts using the internet on at-the-time unresolved questions. We show results benchmarking agent and chain-of-thought forecasting approaches using several LLMs, including the recently-released Claude 4 models, and demonstrate BTF's ability to track steady forecasting capability progress over time. We intend this to be a living benchmark, with new questions added continually to account for increasing training data cutoff dates. We invite researchers to contact us at hello@futuresearch.ai to utilize our benchmark or tooling for their own research.