# PRISM: PRior-guided Imagination Sampling in world Models

arXiv: https://arxiv.org/abs/2606.07974 · HTML: https://mlsanity.com/paper/2606.07974/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 6 Jun 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 17 of 17)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation](https://mlsanity.com/paper/2407.09829/) (2024)
- [Biased-MPPI: Informing Sampling-Based Model Predictive Control by Fusing Ancillary Controllers](https://mlsanity.com/paper/2401.09241/) (2024)
- [Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation](https://mlsanity.com/paper/2504.05225/) (2025)
- [Policy-Guided World Model Planning for Language-Conditioned Visual Navigation](https://mlsanity.com/paper/2603.25981/) (2026)

## Built on by (most cited first, 8 of 8)

- [INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models](https://mlsanity.com/paper/2607.26056/) (2026)
- [SAGE: Subgoal-Conditioned Action Generation for Latent World Model Planning](https://mlsanity.com/paper/2607.17973/) (2026)
- [Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models](https://mlsanity.com/paper/2607.23602/) (2026)
- [Representation World Model: Learning States, Transition and Executable Plans in Representation](https://mlsanity.com/paper/2609.29171/) (2026)
- [FlexiWorld: Learning and Planning via Flexible Action Chunks Across Multiple Time Scales](https://mlsanity.com/paper/2609.35138/) (2026)
- [Control-Geometry Straightening for Sampling-Based Latent Planning](https://mlsanity.com/paper/2609.35603/) (2026)
- [Beyond a single latent space: a dual-latent world model for long-horizon planning](https://mlsanity.com/paper/2609.37644/) (2026)
- [LeWAM: End-to-End World and Action Modeling with JEPAs](https://mlsanity.com/paper/2610.12818/) (2026)

## Abstract (arXiv)

A learned world model provides a powerful physical intuition for evaluating future states. But its effectiveness in continuous control also depends critically on how candidate actions are generated for model-based planning. Rather than solely asking how accurately a model can simulate the future, we ask: which candidate actions are worth evaluating in the first place? Existing planners typically search arbitrarily or use expert demonstrations only to initialize a sampling mean, discarding the expert's state-conditioned confidence. Properly guiding this search requires a robust action prior, yet current approaches often rely on independent visual encoders or large-scale VLMs to obtain one. We argue that this architectural bloat is unnecessary: the exact same data - and the learned representations of the world model itself - inherently encode the agent's action intuition. We introduce PRISM, a task-agnostic framework that extracts both from a single dataset while maintaining strict architectural simplicity. Building on a standard JEPA-style latent world model, PRISM attaches a lightweight MLP directly to its frozen encoder to predict a state-conditioned Gaussian prior. At plan time, PRISM fuses this prior into the planner's sampling distribution via a precision-weighted Product-of-Gaussians update. This parameter-free, closed-form integration steers the sampling process, making the prior confident where it is and ceding control where it is not. PRISM improves success rates by 35 percentage points over vanilla world-model-based MPC on Cube and 32 percentage points on PushT, without introducing significant inference overhead.