# How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2110.03608 · HTML: https://mlsanity.com/paper/2110.03608/

## Key facts

- Cited by 6 papers in the map (data through 7 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Hierarchical VAEs Know What They Don't Know](https://mlsanity.com/paper/2102.08248/) (2021)
- [VAEs in the Presence of Missing Data](https://mlsanity.com/paper/2006.05301/) (2020)
- [Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data](https://mlsanity.com/paper/1805.11264/) (2018)
- [Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learning](https://mlsanity.com/paper/1911.12851/) (2019)
- [GeomCA: Geometric Evaluation of Data Representations](https://mlsanity.com/paper/2105.12486/) (2021)

## Built on by (most cited first, 6 of 6)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Geometric Multimodal Contrastive Representation Learning](https://mlsanity.com/paper/2202.03390/) (2022)
- [CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning](https://mlsanity.com/paper/2602.04920/) (2026)
- [Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/2505.04619/) (2025)
- [Sense, Imagine, Act: Multimodal Perception Improves Model-Based Reinforcement Learning for Head-to-Head Autonomous Racing](https://mlsanity.com/paper/2305.04750/) (2023)
- [Perceive, Represent, Generate: Translating Multimodal Information to Robotic Motion Trajectories](https://mlsanity.com/paper/2204.03051/) (2022)

## Abstract (arXiv)

This work addresses the problem of sensing the world: how to learn a multimodal representation of a reinforcement learning agent's environment that allows the execution of tasks under incomplete perceptual conditions. To address such problem, we argue for hierarchy in the design of representation models and contribute with a novel multimodal representation model, MUSE. The proposed model learns hierarchical representations: low-level modality-specific representations, encoded from raw observation data, and a high-level multimodal representation, encoding joint-modality information to allow robust state estimation. We employ MUSE as the sensory representation model of deep reinforcement learning agents provided with multimodal observations in Atari games. We perform a comparative study over different designs of reinforcement learning agents, showing that MUSE allows agents to perform tasks under incomplete perceptual experience with minimal performance loss. Finally, we evaluate the performance of MUSE in literature-standard multimodal scenarios with higher number and more complex modalities, showing that it outperforms state-of-the-art multimodal variational autoencoders in single and cross-modality generation.