# Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament

arXiv: https://arxiv.org/abs/2310.13014 · HTML: https://mlsanity.com/paper/2310.13014/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [A Theory for Emergence of Complex Skills in Language Models](https://mlsanity.com/paper/2307.15936/) (2023)
- [Diminished Diversity-of-Thought in a Standard Large Language Model](https://mlsanity.com/paper/2302.07267/) (2023)
- [When is a crowd wise?](https://mlsanity.com/paper/1406.7563/) (2014)
- [Asking Better Questions -- The Art and Science of Forecasting: A mechanism for truer answers to high-stakes questions](https://mlsanity.com/paper/2303.18006/) (2023)

## Built on by (most cited first, 24 of 24)

- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law](https://mlsanity.com/paper/2402.00795/) (2024)
- [Consistency Checks for Language Model Forecasters](https://mlsanity.com/paper/2412.18544/) (2024)
- [AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy](https://mlsanity.com/paper/2402.07862/) (2024)
- [Predicting Empirical AI Research Outcomes with Language Models](https://mlsanity.com/paper/2506.00794/) (2025)
- [Can Language Models Use Forecasting Strategies?](https://mlsanity.com/paper/2406.04446/) (2024)
- [Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AI](https://mlsanity.com/paper/2312.06941/) (2023)
- [OpenEP: Open-Ended Future Event Prediction](https://mlsanity.com/paper/2408.06578/) (2024)
- [Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material)](https://mlsanity.com/paper/2510.24303/) (2025)
- [Building Social World Models with Large Language Models](https://mlsanity.com/paper/2606.11482/) (2026)
- [Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents](https://mlsanity.com/paper/2605.00420/) (2026)
- [Large Language Models for Predictive Analysis: How Far Are They?](https://mlsanity.com/paper/2505.17149/) (2025)
- [Argumentatively Coherent Judgmental Forecasting](https://mlsanity.com/paper/2507.23163/) (2025)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [Analytica: Soft Propositional Reasoning for Robust and Scalable LLM-Driven Analysis](https://mlsanity.com/paper/2604.23072/) (2026)
- [Detecting Future-related Contexts of Entity Mentions](https://mlsanity.com/paper/2502.15332/) (2025)
- [Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry](https://mlsanity.com/paper/2607.01661/) (2026)
- [WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament](https://mlsanity.com/paper/2608.04008/) (2026)
- [Future Querying: Can LLMs Serve as Implicit Medical World Models?](https://mlsanity.com/paper/2608.23248/) (2026)
- [From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting](https://mlsanity.com/paper/2609.05905/) (2026)
- [Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets](https://mlsanity.com/paper/2609.32885/) (2026)
- [Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts](https://mlsanity.com/paper/2609.33257/) (2026)

## Abstract (arXiv)

Accurately predicting the future would be an important milestone in the capabilities of artificial intelligence. However, research on the ability of large language models to provide probabilistic predictions about future events remains nascent. To empirically test this ability, we enrolled OpenAI's state-of-the-art large language model, GPT-4, in a three-month forecasting tournament hosted on the Metaculus platform. The tournament, running from July to October 2023, attracted 843 participants and covered diverse topics including Big Tech, U.S. politics, viral outbreaks, and the Ukraine conflict. Focusing on binary forecasts, we show that GPT-4's probabilistic forecasts are significantly less accurate than the median human-crowd forecasts. We find that GPT-4's forecasts did not significantly differ from the no-information forecasting strategy of assigning a 50% probability to every question. We explore a potential explanation, that GPT-4 might be predisposed to predict probabilities close to the midpoint of the scale, but our data do not support this hypothesis. Overall, we find that GPT-4 significantly underperforms in real-world predictive tasks compared to median human-crowd forecasts. A potential explanation for this underperformance is that in real-world forecasting tournaments, the true answers are genuinely unknown at the time of prediction; unlike in other benchmark tasks like professional exams or time series forecasting, where strong performance may at least partly be due to the answers being memorized from the training data. This makes real-world forecasting tournaments an ideal environment for testing the generalized reasoning and prediction capabilities of artificial intelligence going forward.