# FinBench: Time-Gated Calibration and Uncertainty Benchmarking for Agentic Financial Forecasting

arXiv: https://arxiv.org/abs/2607.16229 · HTML: https://mlsanity.com/paper/2607.16229/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 7 of 7)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as components of agentic systems that observe, plan, and act. In finance, even "assistive" systems become decision-relevant once their outputs are used to size trades or allocate risk. A key failure mode is the confidence--competence gap: a model that is only slightly better than chance but consistently overconfident will, under typical bet-sizing rules, generate negative long-run growth. Existing benchmarks emphasize semantic understanding or point accuracy, but do not directly test probabilistic calibration under the temporal constraints and non-stationarity that define real markets. We introduce FinBench, a benchmark designed to evaluate calibration and uncertainty quality for financial forecasting in a setting that is (i) strictly time-gated to avoid look-ahead bias and (ii) evaluated with strictly proper scoring rules that penalize hallucinated confidence. FinBench tasks require models to output (a) a probability of positive return and (b) an 80% prediction interval for realized log return; evaluation uses the Brier score and the Winkler interval score, along with skill scores against hard baselines. This paper describes the benchmark specification and reports a small pilot run (one trading day; three liquid tickers; 33 forecasts) as a sanity check of the pipeline. The pilot illustrates how calibration-sensitive metrics distinguish between "confident but fragile" behavior and uncertainty-aware forecasting.