mlsanityData to 7 Oct 2026

Pitfalls in Evaluating Language Model Forecasters

arXiv 2506.00723 ↗First posted 31 May 2025cs.LG cs.AI cs.IRcited by 23

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. In this paper, we argue that, as a community, we should be careful about such conclusions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of temporal leakage, and (2) difficulty in extrapolating from evaluation performance to real-world forecasting. Through systematic analysis and concrete examples from prior work, we demonstrate how evaluation flaws can raise concerns about current and future performance claims. We argue that more rigorous evaluation methodologies are needed to confidently assess the forecasting abilities of LLMs.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 10 of its 24 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

10 of 24 references.

References · 10

  1. 2024Continual Learning for Large Language Models: A Surveycited by 144
  2. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  3. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  4. 2022Forecasting Future World Events with Neural Networkscited by 63
  5. 2024Consistency Checks for Language Model Forecasterscited by 27
  6. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  7. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  8. 2024Reasoning and Tools for Human-Level Forecastingcited by 7
  9. 2024Questionable practices in machine learningcited by 3
  10. 2021Alignment Problems With Current Forecasting Platformscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 17.

23 papers.

Sort:

By area:

  1. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 55
  2. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  3. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  4. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  5. 2025AIA Forecaster: Technical Reportcited by 9
  6. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  7. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  8. 2025Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Askingcited by 2
  9. 2026ForecastBench-Sim: A Simulated-World Forecasting Benchmarkcited by 2
  10. 2026Verifiable Rewards for Calibrated Probabilistic Forecastingcited by 2
  11. 2026Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Studycited by 0
  12. 2026When AI Navigates the Fog of Warcited by 0
11 more
  1. 2026QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervalscited by 0
  2. 2026TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtestingcited by 0
  3. 2026Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetrycited by 0
  4. 2026Global Merger-Arbitrage Forecasting with Language Modelscited by 0
  5. 2026Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scorescited by 0
  6. 2026Competence-Gated Pooling of Language Models and Priors for Event Forecastingcited by 0
  7. 2026Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecastingcited by 0
  8. 2026Can LLMs Predict the Future? A Brier Score Analysis of Prediction Marketscited by 0
  9. 2026CruxBench: A Benchmark of Information Discoverycited by 0
  10. 2026CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inferencecited by 0
  11. 2026Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Changecited by 0