mlsanityData to 7 Oct 2026

Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models

arXiv 2205.13817 ↗First posted 27 May 2022cs.LG cs.AI cs.ROcited by 53

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

World models learn the consequences of actions in vision-based interactive systems. However, in practical scenarios such as autonomous driving, there commonly exists noncontrollable dynamics independent of the action signals, making it difficult to learn effective world models. To tackle this problem, we present a novel reinforcement learning approach named Iso-Dream, which improves the Dream-to-Control framework in two aspects. First, by optimizing the inverse dynamics, we encourage the world model to learn controllable and noncontrollable sources of spatiotemporal changes on isolated state transition branches. Second, we optimize the behavior of the agent on the decoupled latent imaginations of the world model. Specifically, to estimate state values, we roll-out the noncontrollable states into the future and associate them with the current controllable state. In this way, the isolation of dynamics sources can greatly benefit long-horizon decision-making of the agent, such as a self-driving car that can avoid potential risks by anticipating the movement of other vehicles. Experiments show that Iso-Dream is effective in decoupling the mixed dynamics and remarkably outperforms existing approaches in a wide range of visual control and prediction domains.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 53 of its 71 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

53 of 71 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2020Planning to Explore via Self-Supervised World Models12 May
  3. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  4. 2018Recurrent World Models Facilitate Policy Evolution4 Sep
  5. 2017Value Prediction Network11 Jul

Other references · 48

  1. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  2. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2018DeepMind Control Suitecited by 935
  6. 2020Object-Centric Learning with Slot Attentioncited by 865
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  12. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
36 more
  1. 2016Generating Videos with Scene Dynamicscited by 376
  2. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  3. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  4. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  5. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  6. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  7. 2018Stochastic Video Generation with a Learned Priorcited by 244
  8. 2017Stochastic Variational Video Predictioncited by 234
  9. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  10. 2019Recurrent Independent Mechanismscited by 219
  11. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  12. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  13. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  14. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  15. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  16. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  17. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  18. 2020Stochastic Latent Residual Video Predictioncited by 106
  19. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  20. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  21. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  22. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  23. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  24. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  25. 2017Recurrent Environment Simulatorscited by 68
  26. 2020Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Predictioncited by 52
  27. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  28. 2018SDCNet: Video Prediction Using Spatially-Displaced Convolutioncited by 50
  29. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  30. 2019Variational Temporal Abstractioncited by 37
  31. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  32. 2021Learning Semantic-Aware Dynamics for Video Predictioncited by 21
  33. 2020Unsupervised Video Representation Learning by Bidirectional Feature Predictioncited by 17
  34. 2020Unsupervised Video Decomposition using Spatio-temporal Iterative Inferencecited by 12
  35. 2018DYAN: A Dynamical Atoms-Based Network for Video Predictioncited by 11
  36. 2019Goal-Directed Behavior under Variational Predictive Coding: Dynamic Organization of Visual Attention and Working Memorycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 14; 2025: 14; 2026: 13.

53 papers.

Sort:

By area:

  1. 2024Genie: Generative Interactive Environmentscited by 706
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  5. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  6. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  9. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  10. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  11. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  12. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
38 more
  1. 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 27
  2. 2023Learning World Models with Identifiable Factorizationcited by 23
  3. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  4. 2025Object-Centric World Model for Language-Guided Manipulationcited by 16
  5. 2024CarDreamer: Open-Source Learning Platform for World Model based Autonomous Drivingcited by 13
  6. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  7. 2025Dream to Drive with Predictive Individual World Modelcited by 11
  8. 2023SeMAIL: Eliminating Distractors in Visual Imitation via Separated Modelscited by 10
  9. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  10. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  11. 2025A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulatorcited by 7
  12. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  13. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 5
  14. 2023Model-Based Reinforcement Learning with Isolated Imaginationscited by 4
  15. 2024Imagine-2-Drive: Leveraging High-Fidelity World Models via Multi-Modal Diffusion Policiescited by 4
  16. 2026MRI Contrast Enhancement Kinetics World Modelcited by 4
  17. 2023VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Modelcited by 3
  18. 2024Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Followingcited by 3
  19. 2024MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learningcited by 3
  20. 2023Continual Visual Reinforcement Learning with A Life-Long World Modelcited by 2
  21. 2025Ego-centric Learning of Communicative World Models for Autonomous Drivingcited by 2
  22. 2026AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Modelscited by 2
  23. 2026World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied Taskscited by 2
  24. 2026DWM: Separating World Effects from Actions in Latent World Modelscited by 2
  25. 2026Diagnosing JEPA World Models with Action-Conditioned Predictive Consistencycited by 2
  26. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1
  27. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learningcited by 1
  28. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  29. 2026DreamSAC: Learning Hamiltonian World Models via Symmetry Explorationcited by 1
  30. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretrainingcited by 0
  31. 2025MInCo: Mitigating Information Conflicts in Distracted Visual Model-based Reinforcement Learningcited by 0
  32. 2025TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomycited by 0
  33. 2025Vehicle Dynamics Embedded World Models for Autonomous Drivingcited by 0
  34. 2025Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learningcited by 0
  35. 2026Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddingscited by 0
  36. 2026BRICKS-WM: Building Reusability via Interface Composition Kinetics for Structured World Modelscited by 0
  37. 2026Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigationcited by 0
  38. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018Learning Real-World Robot Policies by Dreaming
  2. 2026Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving
  3. 2021Learning to drive from a world on rails
  4. 2022DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction
  5. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
  6. 2026Learning Vision-Language-Action World Models for Autonomous Driving
  7. 2026DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving
  8. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction