mlsanityData to 7 Oct 2026

Forecasting Future World Events with Neural Networks

arXiv 2206.15474 ↗First posted 30 Jun 2022cs.LG cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

Forecasting future world events is a challenging but valuable task. Forecasts of climate, geopolitical conflict, pandemics and economic indicators help shape policy and decision making. In these domains, the judgment of expert humans contributes to the best forecasts. Given advances in language modeling, can these forecasts be automated? To this end, we introduce Autocast, a dataset containing thousands of forecasting questions and an accompanying news corpus. Questions are taken from forecasting tournaments, ensuring high quality, real-world importance, and diversity. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts (avoiding leakage from the future). Motivated by the difficulty of forecasting numbers across orders of magnitude (e.g. global cases of COVID-19 in 2022), we also curate IntervalQA, a dataset of numerical questions and metrics for calibration. We test language models on our forecasting task and find that performance is far below a human expert baseline. However, performance improves with increased model size and incorporation of relevant information from the news corpus. In sum, Autocast poses a novel challenge for large language models and improved performance could bring large practical benefits.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 38 of its 75 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

38 of 75 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
26 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  4. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  6. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  7. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  8. 2018AI safety via debatecited by 384
  9. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  10. 2016Cooperative Inverse Reinforcement Learningcited by 298
  11. 2021Unsolved Problems in ML Safetycited by 291
  12. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  13. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  14. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  15. 2022X-Risk Analysis for AI Researchcited by 60
  16. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  17. 2020Individual Calibration with Randomized Forecastingcited by 50
  18. 2020ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Datacited by 43
  19. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  20. 2022Uncalibrated Models Can Improve Human-AI Collaborationcited by 26
  21. 2020Avoiding Side Effects in Complex Environmentscited by 24
  22. 2021NeuralProphet: Explainable Forecasting at Scalecited by 22
  23. 2021Training Value-Aligned Reinforcement Learning Agents Using a Normative Priorcited by 14
  24. 2019SafeLife 1.0: Exploring Side Effects in Complex Environmentscited by 8
  25. 2020On-The-Fly Information Retrieval Augmentation for Language Modelscited by 3
  26. 2020On Single Point Forecasts for Fat-Tailed Variablescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 12; 2025: 14; 2026: 28.

63 papers.

Sort:

By area:

  1. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  2. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  3. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  4. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  5. 2023Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learningcited by 36
  6. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 36
  7. 2024Consistency Checks for Language Model Forecasterscited by 27
  8. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  9. 2023Natural Selection Favors AIs over Humanscited by 24
  10. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  11. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  12. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
38 more
  1. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  2. 2023AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrievalcited by 18
  3. 2025Predicting Empirical AI Research Outcomes with Language Modelscited by 14
  4. 2024Can Language Models Use Forecasting Strategies?cited by 13
  5. 2023Towards the Scalable Evaluation of Cooperativeness in Language Modelscited by 12
  6. 2023Conditioning Predictive Models: Risks and Strategiescited by 11
  7. 2023Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AIcited by 11
  8. 2024OpenEP: Open-Ended Future Event Predictioncited by 11
  9. 2026PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Datacited by 11
  10. 2025AIA Forecaster: Technical Reportcited by 9
  11. 2026Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefscited by 9
  12. 2024Event-level Knowledge Editingcited by 8
  13. 2025FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmarkcited by 8
  14. 2024Reasoning and Tools for Human-Level Forecastingcited by 7
  15. 2023Drafting Event Schemas using Language Modelscited by 6
  16. 2025Superintelligence Strategy: Expert Versioncited by 6
  17. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  18. 2025Do Large Language Models Know What They Don't Know? Kalshibench: A New Benchmark for Evaluating Epistemic Calibration via Prediction Marketscited by 5
  19. 2025Do Language Models Update their Forecasts with New Information?cited by 2
  20. 2025From Prediction to Foresight: The Role of AI in Designing Responsible Futurescited by 2
  21. 2026WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoningcited by 2
  22. 2026Reinforcement Learning for LLM-based Event Forecastingcited by 2
  23. 2026What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulnesscited by 2
  24. 2026WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecastingcited by 2
  25. 2026Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensemblescited by 2
  26. 2024Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modelingcited by 1
  27. 2025Anticipatory Evaluation of Language Modelscited by 1
  28. 2025The Wisdom of Deliberating AI Crowds: Does Deliberation Improve LLM-Based Forecasting?cited by 1
  29. 2026All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtestingcited by 1
  30. 2026Harnessing Pre-Resolution Signals for Future Prediction Agentscited by 1
  31. 2026Evaluating Strategic Reasoning in Forecasting Agentscited by 1
  32. 2026LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sportscited by 1
  33. 2026LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecastingcited by 1
  34. 2024SimpleStrat: Diversifying Language Model Generation with Stratificationcited by 0
  35. 2025Detecting Future-related Contexts of Entity Mentionscited by 0
  36. 2026Scalable Delphi: Large Language Models for Structured Risk Estimationcited by 0
  37. 2026StakeBench: Evaluating Language Understanding Grounded in Market Commitmentcited by 0
  38. 2026Decentralized Aggregation of LLM Predictions via Wagering Mechanismscited by 0

50 most cited of 63 All 63 (JSON)