# Latent Geometry Beyond Search: Amortizing Planning in World Models

arXiv: https://arxiv.org/abs/2605.08732 · HTML: https://mlsanity.com/paper/2605.08732/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 23 of its 27 references that are in the map.
- First posted on arXiv 9 May 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 23 of 23)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation](https://mlsanity.com/paper/2412.15109/) (2024)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs](https://mlsanity.com/paper/2512.15692/) (2025)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Planning from Pixels using Inverse Dynamics Models](https://mlsanity.com/paper/2012.02419/) (2020)
- [Joint Embedding Predictive Architectures Focus on Slow Features](https://mlsanity.com/paper/2211.10831/) (2022)
- [Grounding Video Models to Actions through Goal Conditioned Exploration](https://mlsanity.com/paper/2411.07223/) (2024)
- [Latent Diffusion Planning for Imitation Learning](https://mlsanity.com/paper/2504.16925/) (2025)
- [When Does Predictive Inverse Dynamics Outperform Behavior Cloning?](https://mlsanity.com/paper/2601.21718/) (2026)
- [On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning](https://mlsanity.com/paper/2602.02762/) (2026)

## Built on by (most cited first, 15 of 15)

- [INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models](https://mlsanity.com/paper/2607.26056/) (2026)
- [Is Forward Prediction Enough? Physical State Grounding for JEPA World Models](https://mlsanity.com/paper/2608.06799/) (2026)
- [QQWorld: Quantile-Quantile Matching for World Model Regularization](https://mlsanity.com/paper/2607.28415/) (2026)
- [Music-JEPA: Learning a World Model of Sound from Action](https://mlsanity.com/paper/2607.22000/) (2026)
- [ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Models](https://mlsanity.com/paper/2609.05461/) (2026)
- [LePlanner: An Iterative Amortized Controller For World Models](https://mlsanity.com/paper/2609.13845/) (2026)
- [FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning](https://mlsanity.com/paper/2609.22816/) (2026)
- [Representation World Model: Learning States, Transition and Executable Plans in Representation](https://mlsanity.com/paper/2609.29171/) (2026)
- [Think Fast, Plan Selectively: Adaptive Deliberation for Efficient Data-Driven MPC](https://mlsanity.com/paper/2609.32591/) (2026)
- [Adaptive Latent Capacity for World Models](https://mlsanity.com/paper/2609.32921/) (2026)
- [FlexiWorld: Learning and Planning via Flexible Action Chunks Across Multiple Time Scales](https://mlsanity.com/paper/2609.35138/) (2026)
- [NeuronDiscover: Agent-in-Twin for Mechanistic Discovery in Neuronal Microenvironments with World Action Models](https://mlsanity.com/paper/2609.35338/) (2026)
- [Beyond a single latent space: a dual-latent world model for long-horizon planning](https://mlsanity.com/paper/2609.37644/) (2026)
- [Directed Temporal Representations for Offline Visual Control](https://mlsanity.com/paper/2610.08960/) (2026)
- [LeWAM: End-to-End World and Action Modeling with JEPAs](https://mlsanity.com/paper/2610.12818/) (2026)

## Abstract (arXiv)

Modern vision-based world models can represent observations as compact yet expressive latent manifolds, but fast goal-oriented planning in these spaces remains challenging. This raises a central question: when does a learned representation simplify control, rather than merely enabling prediction? We study this question in a pretrained LeWorldModel, whose latent geometry is regularized for smoothness and uniformity. Our key insight is that, under such geometry, planning can be amortized into a latent inverse-dynamics mapping instead of requiring online search. We therefore replace iterative planning with a lightweight Goal-Conditioned Inverse Dynamics Model (GC-IDM) that maps the current latent state, goal latent state, and remaining horizon directly to the next action. Empirically, across four benchmark environments spanning navigation, contact-rich manipulation, and continuous control, our controller matches or exceeds CEM in seven of eight environment-protocol settings while reducing per-decision cost by 100-130x. A broader sweep over test-time planners (CEM, MPPI, iCEM, and gradient-based methods) shows that this result is not specific to a particular optimizer. These findings suggest that much of the structure recovered by test-time planning is already locally encoded in the latent representation. More broadly, our results indicate that sufficiently structured latent spaces can shift part of the planning burden from online optimization to learned inference. Our code is publicly available at https://github.com/hdnndh/Latent-Geometry-Beyond-Search-Amortizing-Planning-in-World-Models .