mlsanityData to 7 Oct 2026

Approaching Human-Level Forecasting with Language Models

arXiv 2402.18563 ↗First posted 28 Feb 2024cs.LG cs.AI cs.CLcited by 102

Not on the map. Why?

Abstract (arXiv)

Forecasting future events is important for policy and decision making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters, and in some settings surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions at scale and help to inform institutional decision making.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

32 of 53 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  6. 2024Mixtral of Expertscited by 1,785
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
20 more
  1. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  2. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  3. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  4. 2023A decoder-only foundation model for time-series forecastingcited by 650
  5. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  6. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  9. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  10. 2021Unsolved Problems in ML Safetycited by 291
  11. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  12. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  13. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  14. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  15. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  16. 2022Forecasting Future World Events with Neural Networkscited by 63
  17. 2020ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Datacited by 43
  18. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24
  19. 2023AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrievalcited by 18
  20. 2023Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AIcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 23; 2026: 62.

102 papers.

Sort:

By area:

  1. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  2. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  3. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  4. 2024Argumentative Large Language Models for Explainable and Contestable Claim Verificationcited by 36
  5. 2024Consistency Checks for Language Model Forecasterscited by 27
  6. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  7. 2024Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulationscited by 20
  8. 2025Bench to the Future: A Pastcasting Benchmark for Forecasting Agentscited by 20
  9. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  10. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  11. 2025Predicting Empirical AI Research Outcomes with Language Modelscited by 14
  12. 2024Can Language Models Use Forecasting Strategies?cited by 13
38 more
  1. 2025LLMs Can Teach Themselves to Better Predict the Futurecited by 13
  2. 2024Real-time Factuality Assessment from Adversarial Feedbackcited by 12
  3. 2024OpenEP: Open-Ended Future Event Predictioncited by 11
  4. 2026PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Datacited by 11
  5. 2025AIA Forecaster: Technical Reportcited by 9
  6. 2026Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefscited by 9
  7. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  8. 2025Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material)cited by 8
  9. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  10. 2024Zero-shot Safety Prediction for Autonomous Robots with Foundation World Modelscited by 6
  11. 2025Evaluating LLMs on Real-World Forecasting Against Expert Forecasterscited by 6
  12. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  13. 2026Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoffcited by 6
  14. 2025Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoningcited by 5
  15. 2025Do Large Language Models Know What They Don't Know? Kalshibench: A New Benchmark for Evaluating Epistemic Calibration via Prediction Marketscited by 5
  16. 2025Adaptively profiling models with task elicitationcited by 4
  17. 2026Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agentscited by 4
  18. 2024DeFine: Decision-Making with Analogical Reasoning over Factor Profilescited by 3
  19. 2025OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Datacited by 3
  20. 2026Automating Forecasting Question Generation and Resolution for AI Evaluationcited by 3
  21. 2026The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmissioncited by 3
  22. 2026Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecastingcited by 3
  23. 2026Dutch Books for Language Modelscited by 3
  24. 2024Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuningcited by 2
  25. 2025Argumentatively Coherent Judgmental Forecastingcited by 2
  26. 2025Do Language Models Update their Forecasts with New Information?cited by 2
  27. 2025Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Askingcited by 2
  28. 2026Coordination as an Architectural Layer for LLM-Based Multi-Agent Systemscited by 2
  29. 2026OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Maskingcited by 2
  30. 2026Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluationcited by 2
  31. 2026Reinforcement Learning for LLM-based Event Forecastingcited by 2
  32. 2026ForecastBench-Sim: A Simulated-World Forecasting Benchmarkcited by 2
  33. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecastingcited by 2
  34. 2026Verifiable Rewards for Calibrated Probabilistic Forecastingcited by 2
  35. 2026What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulnesscited by 2
  36. 2026Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensemblescited by 2
  37. 2024Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modelingcited by 1
  38. 2025Tell Me Why: Incentivizing Explanationscited by 1

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Reasoning and Tools for Human-Level Forecasting
  2. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection
  3. 2025Can Competition Enhance the Proficiency of Agents Powered by Large Language Models in the Realm of News-driven Time Series Forecasting?
  4. 2026TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting
  5. 2024A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting
  6. 2026Rethinking the Role of LLMs in Time Series Forecasting
  7. 2025Conversational Time Series Foundation Models: Towards Explainable and Effective Forecasting
  8. 2024TimeRAG: BOOSTING LLM Time Series Forecasting via Retrieval-Augmented Generation