mlsanityData to 14 Oct 2026

CF-JEPA: Improving Robustness of JEPA World Models via Controllability Factorization

arXiv 2610.00727 ↗First posted 30 Sep 2026cs.RO cs.LGcited by 0

On the map: World models → Latent predictive world models also fits: Embodied and robot world models

Abstract (arXiv)

Controlling an agent with vision requires being able to separate useful information from irrelevant background information. JEPA-style latent world models seem like a natural approach for this, as they do not perform pixel-level reconstruction; however, they are still sensitive to these distractor signals and experience latent collapse. In this work, we introduce Controllability Factorized JEPA (CF-JEPA), a JEPA-style world model which splits the latent space into controllable and uncontrollable subspaces. This factorization allows us to capture all the distractor information into the uncontrollable region, while we use the control-relevant latent information for our task. With this, we show comparable performance across 2D and 3D control tasks under nominal conditions and improved performance under distracted conditions, where CF-JEPA is the only model that does not experience latent collapse. We also validate our model under distracted conditions for a simulated robot task, highlighting the practical application of such a scheme.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 36 of its 48 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

36 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models18 Aug
  2. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamics18 Jun
  3. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels13 Mar
  4. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning7 Nov
  5. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan

Other references · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2023Mastering Diverse Domains through World Modelscited by 1,253
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2020Mastering Atari with Discrete World Modelscited by 1,007
  7. 2018DeepMind Control Suitecited by 956
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  9. 2018Behavioral Cloning from Observationcited by 492
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  11. 2022Transformers are Sample-Efficient World Modelscited by 328
  12. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
19 more
  1. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  3. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  4. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  5. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  6. 2018State Representation Learning for Control: An Overviewcited by 127
  7. 2018Zero-Shot Visual Imitationcited by 117
  8. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  9. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 74
  10. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  11. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  12. 2021Learning Task Informed Abstractionscited by 58
  13. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 31
  14. 2022DreamingV2: Reinforcement Learning with Discrete World Models without Reconstructioncited by 29
  15. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  16. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 20
  17. 2025stable-pretraining-v1: Foundation Model Research Made Simplecited by 10
  18. 2024ImDy: Human Inverse Dynamics from Imitated Observationscited by 9
  19. 2025PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policiescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning
  2. 2026LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
  3. 2026On the Identifiability of Controlled World Models
  4. 2026Is Forward Prediction Enough? Physical State Grounding for JEPA World Models
  5. 2021Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models
  6. 2026ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Models
  7. 2026Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models
  8. 2026Do-JEPA: From Masking to Intervention in Latent World Models