# Sensorimotor World Models: Perception for Action via Inverse Dynamics

arXiv: https://arxiv.org/abs/2606.20104 · HTML: https://mlsanity.com/paper/2606.20104/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 30 of 43)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Scalable methods for computing state similarity in deterministic Markov Decision Processes](https://mlsanity.com/paper/1911.09291/) (2019)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)

## Built on by (most cited first, 20 of 20)

- [INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models](https://mlsanity.com/paper/2607.26056/) (2026)
- [PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models](https://mlsanity.com/paper/2608.05720/) (2026)
- [QQWorld: Quantile-Quantile Matching for World Model Regularization](https://mlsanity.com/paper/2607.28415/) (2026)
- [No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models](https://mlsanity.com/paper/2608.17542/) (2026)
- [ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games](https://mlsanity.com/paper/2607.26712/) (2026)
- [Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Study](https://mlsanity.com/paper/2606.30068/) (2026)
- [Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection](https://mlsanity.com/paper/2608.06706/) (2026)
- [Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning](https://mlsanity.com/paper/2609.03565/) (2026)
- [MotionJEPA: Preventing Temporal Feature Collapse by Capturing Visual Changes in Latent Space](https://mlsanity.com/paper/2609.23881/) (2026)
- [Adaptive Latent Capacity for World Models](https://mlsanity.com/paper/2609.32921/) (2026)
- [Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State](https://mlsanity.com/paper/2609.33497/) (2026)
- [Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning](https://mlsanity.com/paper/2609.33595/) (2026)
- [Control-Geometry Straightening for Sampling-Based Latent Planning](https://mlsanity.com/paper/2609.35603/) (2026)
- [Bilinear World Models: Learning Representations with Structured Dynamics for Efficient Control](https://mlsanity.com/paper/2609.36305/) (2026)
- [CF-JEPA: Improving Robustness of JEPA World Models via Controllability Factorization](https://mlsanity.com/paper/2610.00727/) (2026)
- [Keeping JEPA World Models Plannable When Little of the Frame Moves](https://mlsanity.com/paper/2610.03137/) (2026)
- [Action-Consequence Alignment for Reliable Planning and Self-Improving in Latent World Models](https://mlsanity.com/paper/2610.04539/) (2026)
- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning](https://mlsanity.com/paper/2610.06805/) (2026)
- [Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models](https://mlsanity.com/paper/2610.07540/) (2026)
- [LeWAM: End-to-End World and Action Modeling with JEPAs](https://mlsanity.com/paper/2610.12818/) (2026)

## Abstract (arXiv)

Perception for action suggests that representations of the world should be shaped not by visual fidelity alone, but by their relevance for actions. At the same time, latent JEPA-style world models advocate learning compact predictive states from high-dimensional observations to facilitate the prediction of future states, but end-to-end training of these models is nontrivial because representations may collapse if our only goal is to construct a latent state that is easy to predict. We introduce a sensorimotor world model (SMWM): a latent world model trained end-to-end with inverse dynamics regularization. This single regularizer addresses both issues: it prevents representation collapse and induces action-aligned representations. By forcing latent states to preserve information about the action underlying a transition, it biases the model toward the controllable degrees of freedom of the environment while discarding uncontrollable distractors. This yields stable latent world models trained from offline, reward-free trajectories, without frozen encoders, exponential moving averages, or complex latent regularizers. Empirically, SMWM learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.