# CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?

arXiv: https://arxiv.org/abs/2608.16829 · HTML: https://mlsanity.com/paper/2608.16829/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 18 of 18)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [On GANs and GMMs](https://mlsanity.com/paper/1805.12462/) (2018)
- [Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos](https://mlsanity.com/paper/2407.16124/) (2024)
- [WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation](https://mlsanity.com/paper/2605.25874/) (2026)
- [Inference-time Physics Alignment of Video Generative Models with Latent World Models](https://mlsanity.com/paper/2601.10553/) (2026)
- [Consistency-diversity-realism Pareto fronts of conditional image generative models](https://mlsanity.com/paper/2406.10429/) (2024)
- [Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality](https://mlsanity.com/paper/2410.05203/) (2024)
- [TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Models](https://mlsanity.com/paper/2511.13704/) (2025)
- [World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertainty](https://mlsanity.com/paper/2512.05927/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [World Consistency Score: A Unified Metric for Video Generation Quality](https://mlsanity.com/paper/2508.00144/) (2025)
- [How Confident are Video Models? Empowering Video Models to Express their Uncertainty](https://mlsanity.com/paper/2510.02571/) (2025)

## Built on by (most cited first, 3 of 3)

- [PAWBench: How Far Are We from Probabilistically Aligned World Modeling?](https://mlsanity.com/paper/2608.27345/) (2026)
- [Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models](https://mlsanity.com/paper/2610.03154/) (2026)
- [How Does Geometry Enter Generated Motion?](https://mlsanity.com/paper/2610.05135/) (2026)

## Abstract (arXiv)

Video world models approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks score individual generations or compare distributions coarsely over a whole dataset, leaving the fine-grained aleatoric uncertainty of specific phenomena untested. We introduce CaliBench, which scores outcomes in a physically interpretable discrete space - a bin index, a die face, a suit, a colour - rather than a learned feature space such as in FID, so the distance from a known reference distribution is measured directly. We curate outcome spaces whose reference is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a skewed European-roulette colour), enabling an exact calibration test. We decompose performance into two orthogonal axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample. A chi-squared test assesses significance; as calibration is its null hypothesis it can evidence only miscalibration, and at N=32 per cell detects only large deviations. We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each. Models consistently concentrate probability mass on a few outcomes rather than reproducing the reference. Most scene-model combinations are significantly miscalibrated, in the extreme collapsing to one outcome, as Veo 3.1 does on dice. On roulette, generations often leave the ball ambiguously placed, giving several models low scorability. Performance varies by scene: no model dominates all nine. We release the protocol and a metric (mean normalised total variation, mnTV) for comparing new models against our results.