# Sandwich-Residuals: Parameter-Efficient Test-time Adaptation of World Models

arXiv: https://arxiv.org/abs/2609.21740 · HTML: https://mlsanity.com/paper/2609.21740/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 18 Sep 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 7 of 7)

- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [A System-Level View on Out-of-Distribution Data in Robotics](https://mlsanity.com/paper/2212.14020/) (2022)
- [Joint Embedding Predictive Architectures Focus on Slow Features](https://mlsanity.com/paper/2211.10831/) (2022)
- [AdaJEPA: An Adaptive Latent World Model](https://mlsanity.com/paper/2606.32026/) (2026)
- [Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models](https://mlsanity.com/paper/2602.18639/) (2026)
- [DWM: Separating World Effects from Actions in Latent World Models](https://mlsanity.com/paper/2607.18715/) (2026)

## Built on by (most cited first, 1 of 1)

- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)

## Abstract (arXiv)

Latent world models enable planning by predicting the effects of actions in a learned representation space, but their predictions can become unreliable when test-time conditions differ from training. Existing test-time adaptation methods address this by updating parts of the pretrained model, often modifying millions of parameters and requiring a choice of which internal components to adapt. We introduce Sandwich-Residuals, a lightweight alternative that keeps the pretrained world model frozen and learns only small residual corrections around the predictor. The residuals are optimized online using the model's self-supervised prediction error and require no rewards, labels, or source-domain data. Across 21 conditions on the AdaJEPA benchmark, our method achieves 1.3× the success rate of the frozen model while retaining 95% of the performance of the strongest AdaJEPA variant and adapting 97-99% fewer parameters. Under compound shifts, this advantage increases to 1.9× the success rate of the frozen model, while remaining comparable to internal block adaptation. We further demonstrate the same adaptation principle on a DINO-WM model for 3-D manipulation. These results suggest that effective test-time adaptation of world models does not necessarily require modifying their pretrained internal weights.