# Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement

arXiv: https://arxiv.org/abs/2302.09318 · HTML: https://mlsanity.com/paper/2302.09318/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 18 Feb 2023.

## Builds on (most cited first, 22 of 22)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [End-to-End Robotic Reinforcement Learning without Reward Engineering](https://mlsanity.com/paper/1904.07854/) (2019)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis](https://mlsanity.com/paper/1911.05544/) (2019)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Multimodal Residual Learning for Visual QA](https://mlsanity.com/paper/1606.01455/) (2016)
- [Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding](https://mlsanity.com/paper/2005.09202/) (2020)
- [See, Hear, and Read: Deep Aligned Representations](https://mlsanity.com/paper/1706.00932/) (2017)
- [Majority Vote of Diverse Classifiers for Late Fusion](https://mlsanity.com/paper/1404.7796/) (2014)
- [EmoNets: Multimodal deep learning approaches for emotion recognition in video](https://mlsanity.com/paper/1503.01800/) (2015)
- [Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation](https://mlsanity.com/paper/1705.10422/) (2017)
- [Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learning](https://mlsanity.com/paper/1911.12851/) (2019)
- [Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialog](https://mlsanity.com/paper/1805.03257/) (2018)
- [Crossmodal Attentive Skill Learner](https://mlsanity.com/paper/1711.10314/) (2017)
- [Recurrent Dirichlet Belief Networks for Interpretable Dynamic Relational Data Modelling](https://mlsanity.com/paper/2002.10235/) (2020)

## Built on by (most cited first, 1 of 1)

- [BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning](https://mlsanity.com/paper/2403.01483/) (2024)

## Abstract (arXiv)

Many real-world applications require an agent to make robust and deliberate decisions with multimodal information (e.g., robots with multi-sensory inputs). However, it is very challenging to train the agent via reinforcement learning (RL) due to the heterogeneity and dynamic importance of different modalities. Specifically, we observe that these issues make conventional RL methods difficult to learn a useful state representation in the end-to-end training with multimodal information. To address this, we propose a novel multimodal RL approach that can do multimodal alignment and importance enhancement according to their similarity and importance in terms of RL tasks respectively. By doing so, we are able to learn an effective state representation and consequentially improve the RL training process. We test our approach on several multimodal RL domains, showing that it outperforms state-of-the-art methods in terms of learning speed and policy quality.