# Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

arXiv: https://arxiv.org/abs/2609.37470 · HTML: https://mlsanity.com/paper/2609.37470/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators](https://mlsanity.com/paper/2406.13415/) (2024)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Incoherent Probability Judgments in Large Language Models](https://mlsanity.com/paper/2401.16646/) (2024)
- [Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations, and Forecast Rankings](https://mlsanity.com/paper/1503.08195/) (2015)
- [Consistency Checks for Language Model Forecasters](https://mlsanity.com/paper/2412.18544/) (2024)
- [What Are the Odds? Language Models Are Capable of Probabilistic Reasoning](https://mlsanity.com/paper/2406.12830/) (2024)
- [Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It](https://mlsanity.com/paper/2609.26758/) (2026)
- [Language Model Probabilities are Not Calibrated in Numeric Contexts](https://mlsanity.com/paper/2410.16007/) (2024)
- [Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)](https://mlsanity.com/paper/2609.24052/) (2026)
- [Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences](https://mlsanity.com/paper/2609.24965/) (2026)
- [Evaluating Decision Models for Text Annotation in Computational Social Science](https://mlsanity.com/paper/2609.24574/) (2026)
- [Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality](https://mlsanity.com/paper/2609.27607/) (2026)
- [Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding](https://mlsanity.com/paper/2609.27678/) (2026)
- [When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs](https://mlsanity.com/paper/2602.06286/) (2026)
- [Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures](https://mlsanity.com/paper/2609.29429/) (2026)
- [Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration](https://mlsanity.com/paper/2605.27752/) (2026)
- [JevOut: Natural Context Can Flip Decision Models](https://mlsanity.com/paper/2609.30243/) (2026)
- [CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models](https://mlsanity.com/paper/2409.19984/) (2024)
- [Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems](https://mlsanity.com/paper/2608.05015/) (2026)
- [Dutch Books for Language Models](https://mlsanity.com/paper/2609.02797/) (2026)
- [ForecastBench-Sim: A Simulated-World Forecasting Benchmark](https://mlsanity.com/paper/2606.18686/) (2026)
- [When Policies Change Probabilities: Modular Decision-Making for LLM Code Review](https://mlsanity.com/paper/2608.02677/) (2026)

## Built on by (most cited first, 3 of 3)

- [Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges](https://mlsanity.com/paper/2610.18122/) (2026)
- [Jev-ToM: Evaluating Theory of Mind in Typed Decision Models](https://mlsanity.com/paper/2610.18610/) (2026)
- [Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?](https://mlsanity.com/paper/2610.19856/) (2026)

## Abstract (arXiv)

Equivalent probability requests can lead to different decisions even when both reports are valid. We introduce probability contracts, a benchmark that connects exact finite-world posteriors, validated event alignment, interface coherence, and failure-aware decision evaluation. Across four model-interface configurations on 1,000 worlds, Kev has lower aggregate canonical posterior error than Jev but larger complement and coarsening residuals; accuracy ordering varies by stratum. Jev's Event and Choice interfaces change the binary action on 32.8% of valid pairs at defer cost 0.10. Post-hoc analyses show that disagreement certifies only 11-52% of mean binary pair error and does not consistently outperform confidence for selection. An action-region characterization and a standard scoring-rule identity explain averaging's expected Brier guarantee relative to random interface selection, but not a decision-loss guarantee at each cost. The loss contrast takes both signs on a 99-cost grid for every configuration; small penalties where both policies beat deferral have pointwise intervals containing zero. Secondary checks specified before collection include a separate 400-root cohort, where Event/Choice effects remain configuration-dependent. A joint surface-order and answer-ID intervention shifts posttrained probabilities. Both bounded reasoning arms yield no valid probability reports, leaving their probability accuracy undefined. Probability contracts make these distinctions measurable by evaluating event semantics, posterior error, coverage, and decision cost together.