mlsanityData to 7 Oct 2026

Multi-View Dreaming: Multi-View World Model with Contrastive Learning

arXiv 2203.11024 ↗First posted 15 Mar 2022cs.AI cs.RO cs.SYcited by 5

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

In this paper, we propose Multi-View Dreaming, a novel reinforcement learning agent for integrated recognition and control from multi-view observations by extending Dreaming. Most current reinforcement learning method assumes a single-view observation space, and this imposes limitations on the observed data, such as lack of spatial information and occlusions. This makes obtaining ideal observational information from the environment difficult and is a bottleneck for real-world robotics applications. In this paper, we use contrastive learning to train a shared latent space between different viewpoints, and show how the Products of Experts approach can be used to integrate and control the probability distributions of latent states for multiple viewpoints. We also propose Multi-View DreamingV2, a variant of Multi-View Dreaming that uses a categorical distribution to model the latent state instead of the Gaussian distribution. Experiments show that the proposed method outperforms simple extensions of existing methods in a realistic robot control task.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 25 of its 29 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

25 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning6 Jul
  2. 2020Mastering Atari with Discrete World Models5 Oct
  3. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction29 Jul
  4. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  5. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 20

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2018DeepMind Control Suitecited by 935
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  7. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  8. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  9. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  10. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  11. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  12. 2021Generalized Multimodal ELBOcited by 86
8 more
  1. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  2. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  3. 2019Variational Temporal Abstractioncited by 37
  4. 2018Learning Actionable Representations from Visual Observationscited by 35
  5. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  6. 2020PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inferencecited by 22
  7. 2020Contrastive Variational Reinforcement Learning for Complex Observationscited by 15
  8. 2019Multi-View Reinforcement Learningcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 1.

5 papers.

  1. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  2. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  3. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  4. 2023Robust Multi-agent Communication via Multi-view Message Certificationcited by 3
  5. 2023World-Model-Based Control for Industrial box-packing of Multiple Objects using NewtonianVAEcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction
  2. 2018Learning Real-World Robot Policies by Dreaming
  3. 2025Multimodal Dreaming: A Global Workspace Approach to World Model-Based Reinforcement Learning
  4. 2026DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation
  5. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models
  6. 2024CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learning
  7. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
  8. 2026Probabilistic Dreaming for World Models