# Do Language Models Update their Forecasts with New Information?

arXiv: https://arxiv.org/abs/2509.23936 · HTML: https://mlsanity.com/paper/2509.23936/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [Mean Absolute Percentage Error for regression models](https://mlsanity.com/paper/1605.02541/) (2016)
- [ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data](https://mlsanity.com/paper/2005.00792/) (2020)
- [MIRAI: Evaluating LLM Agents for Event Forecasting](https://mlsanity.com/paper/2407.01231/) (2024)
- [Time-R1: Towards Comprehensive Temporal Reasoning in LLMs](https://mlsanity.com/paper/2505.13508/) (2025)
- [Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?](https://mlsanity.com/paper/2402.16102/) (2024)
- [FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark](https://mlsanity.com/paper/2502.19676/) (2025)
- [TCP: a Benchmark for Temporal Constraint-Based Planning](https://mlsanity.com/paper/2505.19927/) (2025)
- [PRobELM: Plausibility Ranking Evaluation for Language Models](https://mlsanity.com/paper/2404.03818/) (2024)

## Built on by (most cited first, 2 of 2)

- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents](https://mlsanity.com/paper/2609.28876/) (2026)

## Abstract (arXiv)

Prior work has largely treated future event prediction as a static task, failing to consider how forecasts should evolve as new evidence emerges. To address this gap, we introduce EvolveCast, a framework for evaluating whether large language models (LLMs) appropriately revise their predictions in response to new information. In particular, EvolveCast assesses whether models adjust their forecasts when presented with evidence released after their training cutoff, using human forecasters as a comparative reference for prediction shifts and confidence calibration. We find that while models often track the correct update direction, their quantitative shifts remain heavily under-responsive. Furthermore, both verbalized and logits-based confidence estimates are poorly calibrated compared to the human reference standard. These findings suggest current LLMs are fundamentally limited in their response to new evidence; models treat new information as mere retrieval context rather than as variables that dynamically shift posterior probabilities. EvolveCast highlights the critical need for robust mechanisms to incorporate external knowledge into dynamic belief updating.