mlsanityData to 7 Oct 2026

ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

arXiv 2409.19839 ↗First posted 30 Sep 2024cs.LG cs.AI cs.CLcited by 104

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. To address this gap, we introduce ForecastBench: a dynamic benchmark that evaluates the accuracy of ML systems on an automatically generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, ForecastBench is comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMs on a random subset of questions from the benchmark (N=200). While LLMs have achieved super-human performance on many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM (p-value <0.001). We display system and human scores in a public leaderboard at www.forecastbench.org.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

37 of 70 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  6. 2024Mixtral of Expertscited by 1,785
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  9. 2023A decoder-only foundation model for time-series forecastingcited by 650
  10. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  11. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  12. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
25 more
  1. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2022Data Contamination: From Memorization to Exploitationcited by 234
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  5. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  6. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  7. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  8. 2023What's In My Big Data?cited by 129
  9. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  10. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  11. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  12. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  13. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  14. 2022Forecasting Future World Events with Neural Networkscited by 63
  15. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  16. 2023An Open Source Data Contamination Report for Large Language Modelscited by 47
  17. 2020ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Datacited by 43
  18. 2023Data Contamination Through the Lens of Timecited by 39
  19. 2024How predictable is language model benchmark performance?cited by 36
  20. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24
  21. 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
  22. 2023AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrievalcited by 18
  23. 2023Artificial Intelligence Index Report 2023cited by 13
  24. 2024Can Language Models Use Forecasting Strategies?cited by 13
  25. 2023Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AIcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 80.

104 papers.

Sort:

By area:

  1. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  2. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  3. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 36
  4. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Informationcited by 25
  5. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  6. 2025Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarkingcited by 21
  7. 2025Bench to the Future: A Pastcasting Benchmark for Forecasting Agentscited by 20
  8. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  9. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  10. 2025Predicting Empirical AI Research Outcomes with Language Modelscited by 14
  11. 2025LLMs Can Teach Themselves to Better Predict the Futurecited by 13
  12. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
38 more
  1. 2026PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Datacited by 11
  2. 2025Inferring Events from Time Series using Language Modelscited by 10
  3. 2025AIA Forecaster: Technical Reportcited by 9
  4. 2025FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmarkcited by 8
  5. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  6. 2026FutureX-Pro: Extending Future Prediction to High-Value Vertical Domainscited by 8
  7. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  8. 2026Measuring AI R&D Automationcited by 7
  9. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  10. 2025Evaluating LLMs on Real-World Forecasting Against Expert Forecasterscited by 6
  11. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  12. 2026Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoffcited by 6
  13. 2025CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrencycited by 5
  14. 2025Do Large Language Models Know What They Don't Know? Kalshibench: A New Benchmark for Evaluating Epistemic Calibration via Prediction Marketscited by 5
  15. 2026FinDeepForecast: A Live Multi-Agent System for Benchmarking Deep Research Agents in Financial Forecastingcited by 4
  16. 2026Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agentscited by 4
  17. 2025OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Datacited by 3
  18. 2026Automating Forecasting Question Generation and Resolution for AI Evaluationcited by 3
  19. 2026KellyBench: A Benchmark for Long-Horizon Sequential Decision Makingcited by 3
  20. 2026Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecastingcited by 3
  21. 2026Dutch Books for Language Modelscited by 3
  22. 2025Do Language Models Update their Forecasts with New Information?cited by 2
  23. 2025From Prediction to Foresight: The Role of AI in Designing Responsible Futurescited by 2
  24. 2025Agentic AI for Clustering, Relationship Discovery, and Semantic Trading in Prediction Marketscited by 2
  25. 2026Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecastingcited by 2
  26. 2026CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Predictioncited by 2
  27. 2026FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewardscited by 2
  28. 2026Coordination as an Architectural Layer for LLM-Based Multi-Agent Systemscited by 2
  29. 2026OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Maskingcited by 2
  30. 2026Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluationcited by 2
  31. 2026ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgmentcited by 2
  32. 2026WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoningcited by 2
  33. 2026ForecastBench-Sim: A Simulated-World Forecasting Benchmarkcited by 2
  34. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecastingcited by 2
  35. 2026Verifiable Rewards for Calibrated Probabilistic Forecastingcited by 2
  36. 2026What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulnesscited by 2
  37. 2026WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecastingcited by 2
  38. 2025Anticipatory Evaluation of Language Modelscited by 1

50 most cited of 104 All 104 (JSON)