# Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation

arXiv: https://arxiv.org/abs/2609.10954 · HTML: https://mlsanity.com/paper/2609.10954/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 16 of 16)

- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [Finetuning Offline World Models in the Real World](https://mlsanity.com/paper/2310.16029/) (2023)
- [Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection](https://mlsanity.com/paper/2404.07099/) (2024)
- [When to Update Your Model: Constrained Model-based Reinforcement Learning](https://mlsanity.com/paper/2210.08349/) (2022)
- [Learn the Time to Learn: Replay Scheduling in Continual Learning](https://mlsanity.com/paper/2209.08660/) (2022)
- [A Control Theory of Predictability in Latent World Models](https://mlsanity.com/paper/2607.10362/) (2026)
- [When to retrain a machine learning model](https://mlsanity.com/paper/2505.14903/) (2025)
- [Continual Reinforcement Learning by Planning with Online World Models](https://mlsanity.com/paper/2507.09177/) (2025)
- [Overcoming Statistical Bias in Action-Controllable World Models](https://mlsanity.com/paper/2608.04653/) (2026)
- [Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting](https://mlsanity.com/paper/2303.10144/) (2023)
- [Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models](https://mlsanity.com/paper/2608.08982/) (2026)
- [Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback](https://mlsanity.com/paper/2603.04029/) (2026)

## Built on by (most cited first, 1 of 1)

- [When Should an In-Context Learner Expand Its Hypothesis Space?](https://mlsanity.com/paper/2610.09471/) (2026)

## Abstract (arXiv)

Continual world models must decide whether new data justify changing the model. Fixed replay schedules and prediction-error triggers specify when to update, but neither reveals the value of an individual update: one deployment run cannot show how the same model would have performed at that moment had it held its parameters. We introduce the fork ledger, which branches a deployment stream at pre-registered decision points into matched update and hold continuations under common random numbers. It evaluates both continuations on the same episodes and records Delta R = Rupdate - Rhold. Always applying one fixed update mechanism lowers return on all three simulated control tasks: CartPole (-144.0; checkpoint-bootstrap 95% CI [-185.4,-116.1], against a converged return near 650), Walker (-82.8; [-101.1,-61.7]) and Cheetah (-18.6; [-29.0,-6.6]). Divergence is an outcome of applying the update, so the estimand counts every attempted fork; restricted to the 693 of 720 that did not collapse, CartPole and Walker are unchanged in sign (-113.4 and -82.1) and Cheetah becomes unresolved (-3.9; [-17.5,+13.0]). The task is the unit of inference: each contributes 240 attempted forks over five pretrained checkpoints crossed with two drift directions. The ledger makes counterfactual utility observable for a fixed mechanism, allowing triggers to be judged by the updates they select rather than by surprise detection alone.