# Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

arXiv: https://arxiv.org/abs/2606.31630 · HTML: https://mlsanity.com/paper/2606.31630/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 8 of 8)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [Validating Bayesian Inference Algorithms with Simulation-Based Calibration](https://mlsanity.com/paper/1804.06788/) (2018)
- [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019)
- [Bayesian Workflow](https://mlsanity.com/paper/2011.01808/) (2020)
- [LLM-Prior: A Framework for Knowledge-Driven Prior Elicitation and Aggregation](https://mlsanity.com/paper/2508.03766/) (2025)

## Built on by (most cited first, 1 of 1)

- [Large Language Bayes Is Not Reparameterisation-Invariant](https://mlsanity.com/paper/2610.00265/) (2026)

## Abstract (arXiv)

Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be statistically wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization. The right verifier is therefore not a test suite but the Bayesian workflow itself: posterior predictive checks, simulation-based calibration, sampler diagnostics (hat R, divergences, ESS), and held-out predictive density. We study this calibration oracle along three axes. Detection: on a benchmark of 14 misspecification types across 10 model families (200 instances), it flags the bug with AUC 0.97 (88% at 2% FPR when handed the correct reference program, an upper bound) -- and a fully reference-free version that uses no correct program reaches 62--78% (the upper figure from a small automated model search), versus 0% for a unit-test oracle. Repair: used as feedback in an LLM repair loop across fifteen models, calibration significantly outperforms unit-test feedback -- which is itself significantly worse than no feedback at all, a passing test inducing false confidence that suppresses repair -- and improves over no feedback on strong-but-unsaturated models (GPT-5.1 33→92%, Claude 75→100%; paired McNemar, n=228). Reality: on programs LLMs write from scratch for neutral briefs, 15--47% of runnable ones are statistically misspecified (unit tests catch none), and calibration-guided repair significantly beats LLM-as-judge review, a Bayesian-workflow checklist, and data-summary self-debug. Across all three, the lesson is the same: for probabilistic programs, correctness is calibration, not compilation.