mlsanityData to 7 Oct 2026

Evaluating LLMs on Real-World Forecasting Against Expert Forecasters

arXiv 2507.04562 ↗First posted 6 Jul 2025cs.LG cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, but their ability to forecast future events remains understudied. A year ago, large language models struggle to come close to the accuracy of a human crowd. I evaluate state-of-the-art LLMs on 464 forecasting questions from Metaculus, comparing their performance against top forecasters. Frontier models achieve Brier scores that ostensibly surpass the human crowd but still significantly underperform a group of experts.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 10 of its 21 references that are in the map. First posted on arXiv 6 Jul 2025.

Builds on

10 of 21 references.

References · 10

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  3. 2023A decoder-only foundation model for time-series forecastingcited by 650
  4. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 163
  5. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  6. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  7. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 93
  8. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  9. 2024Curating Grounded Synthetic Data with Global Perspectives for Equitable AIcited by 4
  10. 2024Can Base ChatGPT be Used for Forecasting without Additional Optimization?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrencycited by 5
  2. 2025Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Askingcited by 2
  3. 2026Reinforcement Learning for LLM-based Event Forecastingcited by 2
  4. 2026LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applicationscited by 0
  5. 2026Can LLMs Predict the Future? A Brier Score Analysis of Prediction Marketscited by 0
  6. 2026CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inferencecited by 0