mlsanityData to 14 Oct 2026

Keeping JEPA World Models Plannable When Little of the Frame Moves

arXiv 2610.03137 ↗First posted 2 Oct 2026cs.AIcited by 0

On the map: World models → Latent predictive world models also fits: Embodied and robot world models

Abstract (arXiv)

Specifying a goal in language rather than as a goal frame is a natural interface for planning with a latent world model, but testing it needs scenes in which language must discriminate between several objects. We build SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes. On SLIM a LeWM world model that solves PushT succeeds on under 1% of trials, although a scripted controller with simulator state solves every tier. Probes locate the failure in the encoder: its latent is nearly action-insensitive, neither pusher nor object positions can be decoded from it, and rollouts are no better than copying the current latent forward. One inverse-dynamics auxiliary loss, applied to encoder latents and to predicted latents through a shared head discarded at test time, restores every probe and raises success from 0.003 to 0.35 (0.16 on the hard pushing tier, where a goal-agnostic policy scores zero), and improves PushT at twice the trained horizon. Controls attribute the repair to the gradient into the encoder, and a response sweep shows that the vanilla model plans once enough of the frame responds to actions. A cheap action-sensitivity probe, computable without environment access, acts as an empirical necessary condition: all configurations below its threshold failed to plan. On the repaired latent, a small language-goal head plans from sentences without retraining the world model: it reaches 0.84 on navigation (visual-goal oracle 1.00), follows the named zone when it is swapped with a decoy, and degrades gracefully to unseen nouns. A single goal sentence rarely completes a push, but given the push as a sequence of stage sentences the head raises success on the medium and hard pushing tiers from 0.04 to 0.25, on par with the goal-frame oracle, also when the switch between stages is read from the latent alone.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 47 of its 51 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

47 of 51 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning3 Sep
  2. 2026Decision-Metric Alignment in Latent World Models: Diagnostics and Action-Conditioned Objectives for MPC Planning19 Aug
  3. 2026No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models18 Aug
  4. 2026Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency13 Aug
  5. 2026PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models6 Aug
  6. 2026ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games29 Jul
  7. 2026Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control28 Jul
  8. 2026SAGE: Subgoal-Conditioned Action Generation for Latent World Model Planning20 Jul
  9. 2026Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding30 Jun
  10. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamics18 Jun
  11. 2026Latent Goal Prediction from Language for Model-Based Planning26 May
  12. 2026Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations14 May
8 more
  1. 2026Predictive but Not Plannable: RC-aux for Latent World Models8 May
  2. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels13 Mar
  3. 2026Temporal Straightening for Latent Planning12 Mar
  4. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling4 Mar
  5. 2026Causal-JEPA: Learning World Models through Object-Level Latent Masking11 Feb
  6. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?30 Dec
  7. 2024Revisiting Feature Prediction for Learning Visual Representations from Video15 Feb
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan

Other references · 27

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2018World Modelscited by 1,513
  5. 2023Mastering Diverse Domains through World Modelscited by 1,253
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  9. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  10. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  11. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  12. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
15 more
  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  3. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  4. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  5. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  6. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  7. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  8. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  9. 2023Learning to Model the World with Languagecited by 68
  10. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  11. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 20
  12. 2024Multistep Inverse Is Not All You Needcited by 11
  13. 2026DiLA: Disentangled Latent Action World Modelscited by 10
  14. 2026What Makes Video World Model Latents Action-Relevant: Prediction over Reconstructioncited by 9
  15. 2026Policy-Guided World Model Planning for Language-Conditioned Visual Navigationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
  2. 2026One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
  3. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
  4. 2026LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
  5. 2026AcrossWAM1.0:A Modular Latent World-Action Stack for Compact Robot Policies
  6. 2026Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies
  7. 2022Few-shot Subgoal Planning with Language Models
  8. 2026The Planning Limits of Latent World Models