# Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AI

arXiv: https://arxiv.org/abs/2312.06941 · HTML: https://mlsanity.com/paper/2312.06941/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 86 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Temporal Data Meets LLM -- Explainable Financial Time Series Forecasting](https://mlsanity.com/paper/2306.11025/) (2023)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [Leveraging Language Foundation Models for Human Mobility Forecasting](https://mlsanity.com/paper/2209.05479/) (2022)
- [Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament](https://mlsanity.com/paper/2310.13014/) (2023)
- [Translating Human Mobility Forecasting through Natural Language Generation](https://mlsanity.com/paper/2112.11481/) (2021)

## Built on by (most cited first, 11 of 11)

- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Predicting Empirical AI Research Outcomes with Language Models](https://mlsanity.com/paper/2506.00794/) (2025)
- [Can Language Models Use Forecasting Strategies?](https://mlsanity.com/paper/2406.04446/) (2024)
- [Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material)](https://mlsanity.com/paper/2510.24303/) (2025)
- [Large Language Models for Predictive Analysis: How Far Are They?](https://mlsanity.com/paper/2505.17149/) (2025)
- [Argumentatively Coherent Judgmental Forecasting](https://mlsanity.com/paper/2507.23163/) (2025)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [From Prediction to Foresight: The Role of AI in Designing Responsible Futures](https://mlsanity.com/paper/2511.21570/) (2025)
- [Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts](https://mlsanity.com/paper/2609.33257/) (2026)

## Abstract (arXiv)

This study investigates the forecasting accuracy of human experts versus Large Language Models (LLMs) in the retail sector, particularly during standard and promotional sales periods. Utilizing a controlled experimental setup with 123 human forecasters and five LLMs, including ChatGPT4, ChatGPT3.5, Bard, Bing, and Llama2, we evaluated forecasting precision through Mean Absolute Percentage Error. Our analysis centered on the effect of the following factors on forecasters performance: the supporting statistical model (baseline and advanced), whether the product was on promotion, and the nature of external impact. The findings indicate that LLMs do not consistently outperform humans in forecasting accuracy and that advanced statistical forecasting models do not uniformly enhance the performance of either human forecasters or LLMs. Both human and LLM forecasters exhibited increased forecasting errors, particularly during promotional periods and under the influence of positive external impacts. Our findings call for careful consideration when integrating LLMs into practical forecasting processes.