# Pitfalls in Evaluating Language Model Forecasters

arXiv: https://arxiv.org/abs/2506.00723 · HTML: https://mlsanity.com/paper/2506.00723/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 10 of 10)

- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [Consistency Checks for Language Model Forecasters](https://mlsanity.com/paper/2412.18544/) (2024)
- [Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle](https://mlsanity.com/paper/2411.08324/) (2024)
- [PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimation](https://mlsanity.com/paper/2504.01509/) (2025)
- [Reasoning and Tools for Human-Level Forecasting](https://mlsanity.com/paper/2408.12036/) (2024)
- [Questionable practices in machine learning](https://mlsanity.com/paper/2407.12220/) (2024)
- [Alignment Problems With Current Forecasting Platforms](https://mlsanity.com/paper/2106.11248/) (2021)

## Built on by (most cited first, 23 of 23)

- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [Scaling Open-Ended Reasoning to Predict the Future](https://mlsanity.com/paper/2512.25070/) (2025)
- [Evaluating LLMs in Finance Requires Explicit Bias Consideration](https://mlsanity.com/paper/2602.14233/) (2026)
- [AIA Forecaster: Technical Report](https://mlsanity.com/paper/2511.07678/) (2025)
- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts](https://mlsanity.com/paper/2507.19477/) (2025)
- [Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking](https://mlsanity.com/paper/2511.18394/) (2025)
- [ForecastBench-Sim: A Simulated-World Forecasting Benchmark](https://mlsanity.com/paper/2606.18686/) (2026)
- [Verifiable Rewards for Calibrated Probabilistic Forecasting](https://mlsanity.com/paper/2607.00164/) (2026)
- [Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores](https://mlsanity.com/paper/2608.02985/) (2026)
- [TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting](https://mlsanity.com/paper/2605.18843/) (2026)
- [Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Study](https://mlsanity.com/paper/2602.00758/) (2026)
- [When AI Navigates the Fog of War](https://mlsanity.com/paper/2603.16642/) (2026)
- [QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals](https://mlsanity.com/paper/2604.15859/) (2026)
- [Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry](https://mlsanity.com/paper/2607.01661/) (2026)
- [Global Merger-Arbitrage Forecasting with Language Models](https://mlsanity.com/paper/2607.09921/) (2026)
- [When Do Language Models Add Value? Relative Competence in Hybrid Event Forecasting](https://mlsanity.com/paper/2609.12101/) (2026)
- [Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting](https://mlsanity.com/paper/2609.22820/) (2026)
- [Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets](https://mlsanity.com/paper/2609.32885/) (2026)
- [CruxBench: A Benchmark of Information Discovery](https://mlsanity.com/paper/2609.35879/) (2026)
- [CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference](https://mlsanity.com/paper/2609.36689/) (2026)
- [Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Change](https://mlsanity.com/paper/2609.36739/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. In this paper, we argue that, as a community, we should be careful about such conclusions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of temporal leakage, and (2) difficulty in extrapolating from evaluation performance to real-world forecasting. Through systematic analysis and concrete examples from prior work, we demonstrate how evaluation flaws can raise concerns about current and future performance claims. We argue that more rigorous evaluation methodologies are needed to confidently assess the forecasting abilities of LLMs.