# AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy

arXiv: https://arxiv.org/abs/2402.07862 · HTML: https://mlsanity.com/paper/2402.07862/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 108 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [A Theory for Emergence of Complex Skills in Language Models](https://mlsanity.com/paper/2307.15936/) (2023)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [SlimPajama-DC: Understanding Data Combinations for LLM Training](https://mlsanity.com/paper/2309.10818/) (2023)
- [Spear Phishing With Large Language Models](https://mlsanity.com/paper/2305.06972/) (2023)
- [Diminished Diversity-of-Thought in a Standard Large Language Model](https://mlsanity.com/paper/2302.07267/) (2023)
- [Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction](https://mlsanity.com/paper/2303.09224/) (2023)
- [Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament](https://mlsanity.com/paper/2310.13014/) (2023)
- [Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models](https://mlsanity.com/paper/2308.12287/) (2023)
- [Task Supportive and Personalized Human-Large Language Model Interaction: A User Study](https://mlsanity.com/paper/2402.06170/) (2024)
- ["It would work for me too": How Online Communities Shape Software Developers' Trust in AI-Powered Code Generation Tools](https://mlsanity.com/paper/2212.03491/) (2022)

## Built on by (most cited first, 21 of 21)

- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation](https://mlsanity.com/paper/2407.09886/) (2024)
- [Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations](https://mlsanity.com/paper/2403.03407/) (2024)
- [Can Language Models Use Forecasting Strategies?](https://mlsanity.com/paper/2406.04446/) (2024)
- [Measuring AI R&D Automation](https://mlsanity.com/paper/2603.03992/) (2026)
- [Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material)](https://mlsanity.com/paper/2510.24303/) (2025)
- [Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts](https://mlsanity.com/paper/2507.19477/) (2025)
- [Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff](https://mlsanity.com/paper/2601.13717/) (2026)
- [OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data](https://mlsanity.com/paper/2510.15096/) (2025)
- [The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission](https://mlsanity.com/paper/2605.00844/) (2026)
- [Argumentatively Coherent Judgmental Forecasting](https://mlsanity.com/paper/2507.23163/) (2025)
- [From Prediction to Foresight: The Role of AI in Designing Responsible Futures](https://mlsanity.com/paper/2511.21570/) (2025)
- [An LLM-based Delphi Study to Predict GenAI Evolution](https://mlsanity.com/paper/2502.21092/) (2025)
- [The Wisdom of Deliberating AI Crowds: Does Deliberation Improve LLM-Based Forecasting?](https://mlsanity.com/paper/2512.22625/) (2025)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Towards Real-time Adaptation of Embodied Agent in Human-Robot Collaboration](https://mlsanity.com/paper/2412.00435/) (2024)
- [AI Predicts AGI: Leveraging AGI Forecasting and Peer Review to Explore LLMs' Complex Reasoning Capabilities](https://mlsanity.com/paper/2412.09385/) (2024)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [Global Merger-Arbitrage Forecasting with Language Models](https://mlsanity.com/paper/2607.09921/) (2026)

## Abstract (arXiv)

Large language models (LLMs) match and sometimes exceeding human performance in many domains. This study explores the potential of LLMs to augment human judgement in a forecasting task. We evaluate the effect on human forecasters of two LLM assistants: one designed to provide high-quality ("superforecasting") advice, and the other designed to be overconfident and base-rate neglecting, thus providing noisy forecasting advice. We compare participants using these assistants to a control group that received a less advanced model that did not provide numerical predictions or engaged in explicit discussion of predictions. Participants (N = 991) answered a set of six forecasting questions and had the option to consult their assigned LLM assistant throughout. Our preregistered analyses show that interacting with each of our frontier LLM assistants significantly enhances prediction accuracy by between 24 percent and 28 percent compared to the control group. Exploratory analyses showed a pronounced outlier effect in one forecasting item, without which we find that the superforecasting assistant increased accuracy by 41 percent, compared with 29 percent for the noisy assistant. We further examine whether LLM forecasting augmentation disproportionately benefits less skilled forecasters, degrades the wisdom-of-the-crowd by reducing prediction diversity, or varies in effectiveness with question difficulty. Our data do not consistently support these hypotheses. Our results suggest that access to a frontier LLM assistant, even a noisy one, can be a helpful decision aid in cognitively demanding tasks compared to a less powerful model that does not provide specific forecasting advice. However, the effects of outliers suggest that further research into the robustness of this pattern is needed.