mlsanityData to 7 Oct 2026

Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets

arXiv 2609.32885 ↗First posted 26 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We study whether model upgrades improve probability estimates for prediction-market questions. Our Resolved Market Forecasting (RMF) benchmark contains 3,000 resolved binary questions across nine domains, on which we evaluate six Claude and Qwen model variants using a question-only, zero-shot protocol. We assess Brier scores relative to an empirical base-rate predictor, examine their Murphy decomposition, and compare models through paired differences, with results stratified by event category and timing relative to training cutoffs. In the reported post-cutoff stratum, the four Claude models achieve Brier scores of 0.183-0.192, improving on the base-rate reference by 0.024-0.033. Qwen 32B does not significantly outperform that reference, although its paired Brier is 0.024 lower than that of the 7B checkpoint. The evaluated Claude version and tier upgrades yield no significant improvement. Within-model differences across event categories exceed the observed differences among Claude variants. These results show why forecasting scores should be interpreted alongside simple probability baselines and question composition: under this protocol, newer versions or higher model tiers do not consistently produce more accurate probabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

18 of 29 references.

References · 18

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  4. 2022Data Contamination: From Memorization to Exploitationcited by 234
  5. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  6. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  7. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  8. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  9. 2023Data Contamination Through the Lens of Timecited by 39
  10. 2024Consistency Checks for Language Model Forecasterscited by 27
  11. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24
  12. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
6 more
  1. 2014When is a crowd wise?cited by 13
  2. 2025LLMs Can Teach Themselves to Better Predict the Futurecited by 13
  3. 2025AIA Forecaster: Technical Reportcited by 9
  4. 2025Evaluating LLMs on Real-World Forecasting Against Expert Forecasterscited by 6
  5. 2026Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoffcited by 6
  6. 2025Do Large Language Models Know What They Don't Know? Kalshibench: A New Benchmark for Evaluating Epistemic Calibration via Prediction Marketscited by 5

Built on by

Not cited yet.