# When Do Language Models Add Value? Relative Competence in Hybrid Event Forecasting

arXiv: https://arxiv.org/abs/2609.12101 · HTML: https://mlsanity.com/paper/2609.12101/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Pitfalls in Evaluating Language Model Forecasters](https://mlsanity.com/paper/2506.00723/) (2025)
- [LLMs Can Teach Themselves to Better Predict the Future](https://mlsanity.com/paper/2502.05253/) (2025)
- [PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data](https://mlsanity.com/paper/2604.14199/) (2026)
- [PolySwarm: A Multi-Agent Large Language Model Framework for Prediction Market Trading and Latency Arbitrage](https://mlsanity.com/paper/2604.03888/) (2026)
- [Cross-calibration of probabilistic forecasts](https://mlsanity.com/paper/1505.05314/) (2015)
- [Local Prediction Pools](https://mlsanity.com/paper/2112.09073/) (2021)
- [Bayesian Predictive Synthesis with Outcome-Dependent Pools](https://mlsanity.com/paper/1803.01984/) (2018)

## Abstract (arXiv)

When a forecasting system already has a prediction from a market, crowd, or statistical model, when should it use a language model? Standalone accuracy cannot answer this question because a less accurate model may still improve the existing forecast. We study this distinction through relative competence, separating a model's replacement advantage from its complementary value. Under Brier loss (squared error between predicted probabilities and binary outcomes), we characterize when model disagreement can correct external forecast errors and derive the population gain from regime-specific affine pooling. We introduce a competence gate that estimates source weights from resolved outcomes, shrinks regime-specific estimates toward a global solution, and chronologically recalibrates the pooled forecast. Across 4,974 resolved binary questions and four language models, the best standalone model has 24.94% higher Brier loss than the calibrated external forecast, yet the gate improves that reference by 3.35% (95% CI, 1.80 to 4.90%). Model contributions vary across forecasting regimes, with larger corrections in thinner markets and forecasting crowds and smaller corrections in higher-participation settings. The gate also shows no statistically detectable performance difference from more flexible contextual aggregators. These results demonstrate that standalone superiority is not necessary for complementary predictive value and that language models should be evaluated relative to the forecasts already available.