mlsanityData to 14 Oct 2026

What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?

arXiv 2512.24497 ↗First posted 30 Dec 2025cs.AI cs.LG cs.ROcited by 43

On the map: World models → Latent predictive world models

Abstract (arXiv)

A long-standing challenge in AI is to develop agents capable of solving a wide range of physical tasks and generalizing to new, unseen tasks and environments. A popular recent approach involves training a world model from state-action trajectories and subsequently use it with a planning algorithm to solve new tasks. Planning is commonly performed in the input space, but a recent family of methods has introduced planning algorithms that optimize in the learned representation space of the world model, with the promise that abstracting irrelevant details yields more efficient planning. In this work, we characterize models from this family as JEPA-WMs and investigate the technical choices that make algorithms from this class work. We propose a comprehensive study of several key components with the objective of finding the optimal approach within the family. We conducted experiments using both simulated environments and real-world robotic data, and studied how the model architecture, the training objective, and the planning algorithm affect planning success. We combine our findings to propose a model that outperforms two established baselines, DINO-WM and V-JEPA-2-AC, in both navigation and manipulation tasks. Code, data and checkpoints are available at https://github.com/facebookresearch/jepa-wms.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 55 of its 72 references that are in the map. First posted on arXiv 30 Dec 2025.

Builds on

55 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning7 Nov
  2. 2024Revisiting Feature Prediction for Learning Visual Representations from Video15 Feb

Other references · 53

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
38 more
  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  6. 2023Vision Transformers Need Registerscited by 871
  7. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  10. 2024Genie: Generative Interactive Environmentscited by 747
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  12. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  13. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  14. 2019Model-Based Reinforcement Learning for Ataricited by 502
  15. 2023Learning Interactive Real-World Simulatorscited by 458
  16. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  17. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  18. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  19. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  20. 2024Latent Action Pretraining from Videoscited by 321
  21. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  22. 2024Navigation World Modelscited by 272
  23. 2019Understanding and Improving Layer Normalizationcited by 217
  24. 2022Masked World Models for Visual Controlcited by 170
  25. 2019Planning with Goal-Conditioned Policiescited by 149
  26. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  27. 2021Discovering and Achieving Goals via World Modelscited by 131
  28. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  29. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  30. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  31. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  32. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  33. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 58
  34. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 50
  35. 2024Diffusion Model Predictive Controlcited by 36
  36. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  37. 2021The MIT Humanoid Robot: Design, Motion Planning, and Control For Acrobatic Behaviorscited by 32
  38. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 43.

43 papers.

Sort:

By area:

  1. 2026Learning Latent Action World Models In The Wildcited by 47
  2. 2026Hierarchical Planning with Latent World Modelscited by 43
  3. 2026Temporal Straightening for Latent Planningcited by 38
  4. 2026VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Modelscited by 17
  5. 2026A Lightweight Library for Energy-Based Joint-Embedding Predictive Architecturescited by 14
  6. 2026Action100M: A Large-scale Video Action Datasetcited by 12
  7. 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanationcited by 10
  8. 2026A Generalization Theory for JEPA-Based World Modelscited by 5
  9. 2026Latent World Models with Monotone Planning Costs for Image-Goal Navigationcited by 5
  10. 2026BiJEPA: Bi-directional Joint Embedding Predictive Architecture for Symmetric Representation Learningcited by 4
  11. 2026Policy-Guided World Model Planning for Language-Conditioned Visual Navigationcited by 4
  12. 2026Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representationscited by 4
31 more
  1. 2026ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulationcited by 4
  2. 2026Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimagingcited by 4
  3. 2026Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Controlcited by 4
  4. 2026Grounding Generated Videos in Feasible Plans via World Modelscited by 3
  5. 2026What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representationscited by 3
  6. 2026Traj-LeWM: Path-Aware World-Model Planning via Latent Trajectory Costcited by 3
  7. 2026Reward Prediction with Factorized World Statescited by 2
  8. 2026Knowledge, Rules and Their Embeddings: Two Paths towards Neuro-Symbolic JEPAcited by 2
  9. 2026Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Taskscited by 2
  10. 2026The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Modelscited by 2
  11. 2026Grounded World Model for Semantically Generalizable Planningcited by 1
  12. 2026PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignmentcited by 0
  13. 2026Optimizing ML Workload Partitioning between CPUs and CIM Accelerators for Heterogeneous Computingcited by 0
  14. 2026TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulationcited by 0
  15. 2026JEPADepth: Masked Predictive Representation Learning for Self-Supervised Monocular Depth Estimationcited by 0
  16. 2026ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Modelscited by 0
  17. 2026DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulationcited by 0
  18. 2026D-JEPA: A Decision-Aligned Latent World Modelcited by 0
  19. 2026Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Thinkcited by 0
  20. 2026What Must a World Model Distinguish for Planning?cited by 0
  21. 2026Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planningcited by 0
  22. 2026ViBR-WM: Visual Bayesian Regression for World Modelingcited by 0
  23. 2026AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Drivingcited by 0
  24. 2026PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEscited by 0
  25. 2026Beyond a single latent space: a dual-latent world model for long-horizon planningcited by 0
  26. 2026Keeping JEPA World Models Plannable When Little of the Frame Movescited by 0
  27. 2026H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planningcited by 0
  28. 2026Juno: Taming Predictive Latents for Vision-Language-Action Modelscited by 0
  29. 2026Sparse Planning in Visual World Models via Cost Gradientscited by 0
  30. 2026MORE: Learning Motion-Oriented Registers for Driving World Models and Planningcited by 0
  31. 2026Box-WAM: Box Trajectories as Compact State Representations for World Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Value-guided action planning with JEPA world models
  2. 2026Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning
  3. 2026The Planning Limits of Latent World Models
  4. 2026FF-JEPA: Long-Horizon Planning in World Models with Latent Planners
  5. 2026Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control
  6. 2026SCALE: State-Calibrated Latent Embeddings for JEPA Planning in the Right Geometry
  7. 2026Representation World Model: Learning States, Transition and Executable Plans in Representation
  8. 2026LeWAM: End-to-End World and Action Modeling with JEPAs