mlsanityData to 14 Oct 2026

LeWAM: End-to-End World and Action Modeling with JEPAs

arXiv 2610.12818 ↗First posted 8 Oct 2026cs.RO cs.AIcited by 0

On the map: World models → Latent predictive world models also fits: Embodied and robot world models

Abstract (arXiv)

Joint-embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction. Existing JEPA pipelines typically train visual representations and action-conditioned dynamics before adding a control policy or solver. We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics. Compared with prior JEPAs in visual planning, LeWAM improves success rate from 28.6% to 89.7% on contact-rich tasks and from 10.9% to 46.2% on long-horizon tasks, and plans 32.3 times faster. On behavior cloning tasks in simulation and physical robots, LeWAM is also competitive with established methods. Interpretability analyses show that LeWAM learns control-relevant representations and action distributions effectively guiding test-time planning and control.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

46 of 53 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamics18 Jun
  2. 2026PRISM: PRior-guided Imagination Sampling in world Models6 Jun
  3. 2026Latent Geometry Beyond Search: Amortizing Planning in World Models9 May
  4. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels13 Mar
  5. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning7 Nov
  6. 2024Revisiting Feature Prediction for Learning Visual Representations from Video15 Feb
  7. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan
  8. 2022Joint Embedding Predictive Architectures Focus on Slow Features20 Nov
  9. 2022Temporal Difference Learning for Model Predictive Control9 Mar
  10. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2018World Modelscited by 1,513
  9. 2023Mastering Diverse Domains through World Modelscited by 1,253
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  11. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
24 more
  1. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  2. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  3. 2026World Action Models are Zero-shot Policiescited by 465
  4. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  5. 2022Transformers are Sample-Efficient World Modelscited by 328
  6. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  7. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  8. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  9. 2025Unified Video Action Modelcited by 205
  10. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  11. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  12. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  13. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  14. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
  15. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  16. 2026LeRobot: An Open-Source Library for End-to-End Robot Learningcited by 73
  17. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 49
  18. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 31
  19. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 20
  20. 2023Gradient-based Planning with World Modelscited by 19
  21. 2026VISReg: Variance-Invariance-Sketching Regularization for JEPA trainingcited by 16
  22. 2021CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learningcited by 8
  23. 2024D-Cubed: Latent Diffusion Trajectory Optimisation for Dexterous Deformable Manipulationcited by 4
  24. 2025Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LeWAM: A JEPA World Action Model with Diffusion-Steering-Based MPC
  2. 2026Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning
  3. 2026Fast LeWorldModel
  4. 2026JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling
  5. 2026LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
  6. 2026Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Control
  7. 2026H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
  8. 2026Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State