mlsanityData to 7 Oct 2026

Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning

arXiv 2107.02339 ↗First posted 6 Jul 2021cs.LG cs.AI cs.ROcited by 13

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

This work focuses on learning useful and robust deep world models using multiple, possibly unreliable, sensors. We find that current methods do not sufficiently encourage a shared representation between modalities; this can cause poor performance on downstream tasks and over-reliance on specific sensors. As a solution, we contribute a new multi-modal deep latent state-space model, trained using a mutual information lower-bound. The key innovation is a specially-designed density ratio estimator that encourages consistency between the latent codes of each modality. We tasked our method to learn policies (in a self-supervised manner) on multi-modal Natural MuJoCo benchmarks and a challenging Table Wiping task. Experiments show our method significantly outperforms state-of-the-art deep reinforcement learning methods, particularly in the presence of missing observations.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 14 of its 23 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

14 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 13

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  6. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  7. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  8. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  9. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 80
  10. 2020Event-Driven Visual-Tactile Sensing and Learning for Robotscited by 27
  11. 2020Contrastive Variational Reinforcement Learning for Complex Observationscited by 15
  12. 2019Multimodal representation models for prediction and control from partial informationcited by 11
1 more
  1. 2019Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Seriescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 2; 2024: 1; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  2. 2022DreamingV2: Reinforcement Learning with Discrete World Models without Reconstructioncited by 29
  3. 2023Industrial Internet of Things Intelligence Empowering Smart Manufacturing: A Literature Reviewcited by 8
  4. 2022MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communicationcited by 7
  5. 2022Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrationscited by 6
  6. 2022Multi-View Dreaming: Multi-View World Model with Contrastive Learningcited by 5
  7. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  8. 2024Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensorscited by 4
  9. 2021Surrogate-based cross-correlation for particle image velocimetrycited by 2
  10. 2025New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehiclescited by 2
  11. 2022Generalizing Multimodal Variational Methods to Setscited by 0
  12. 2023Latent Emission-Augmented Perspective-Taking (LEAPT) for Human-Robot Interactioncited by 0
1 more
  1. 2026Fuzzy Encoding-Decoding to Improve Spiking Q-Learning Performance in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning
  2. 2026MA-JEPA: Joint-Embedding World Models for Multi-Agent Reinforcement Learning
  3. 2026Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
  4. 2022On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning
  5. 2026SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
  6. 2023Learning World Models with Identifiable Factorization
  7. 2025From Pixels to Cooperation Multi Agent Reinforcement Learning based on Multimodal World Models
  8. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners