# Prompt Engineering Large Language Models' Forecasting Capabilities

arXiv: https://arxiv.org/abs/2506.01578 · HTML: https://mlsanity.com/paper/2506.01578/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 1 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Built on by (most cited first, 5 of 5)

- [Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs](https://mlsanity.com/paper/2604.18576/) (2026)
- [AIA Forecaster: Technical Report](https://mlsanity.com/paper/2511.07678/) (2025)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Assessing mentalization in humans and large language models](https://mlsanity.com/paper/2608.26291/) (2026)
- [From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting](https://mlsanity.com/paper/2609.05905/) (2026)

## Abstract (arXiv)

Large language model performance can be improved in a large number of ways. Many such techniques, like fine-tuning or advanced tool usage, are time-intensive and expensive. Although prompt engineering is significantly cheaper and often works for simpler tasks, it remains unclear whether prompt engineering suffices for more complex domains like forecasting. Here we show that small prompt modifications rarely boost forecasting accuracy beyond a minimal baseline. In our first study, we tested 38 prompts across Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4o, and Llama 3.1 405B. In our second, we introduced compound prompts and prompts from external sources, also including the reasoning models o1 and o1-mini. Our results show that most prompts lead to negligible gains, although references to base rates yield slight benefits. Surprisingly, some strategies showed strong negative effects on accuracy: especially encouraging the model to engage in Bayesian reasoning. These results suggest that, in the context of complex tasks like forecasting, basic prompt refinements alone offer limited gains, implying that more robust or specialized techniques may be required for substantial performance improvements in AI forecasting.