# Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learning

arXiv: https://arxiv.org/abs/1911.12851 · HTML: https://mlsanity.com/paper/1911.12851/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 9 of 9)

- [Geometric Multimodal Contrastive Representation Learning](https://mlsanity.com/paper/2202.03390/) (2022)
- [Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challenges](https://mlsanity.com/paper/2311.18044/) (2023)
- [Semi-supervised Multimodal Representation Learning through a Global Workspace](https://mlsanity.com/paper/2306.15711/) (2023)
- [DareFightingICE Competition: A Fighting Game Sound Design and AI Competition](https://mlsanity.com/paper/2203.01556/) (2022)
- [Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning](https://mlsanity.com/paper/2105.14780/) (2021)
- [How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents](https://mlsanity.com/paper/2110.03608/) (2021)
- [Zero-shot cross-modal transfer of Reinforcement Learning policies through a Global Workspace](https://mlsanity.com/paper/2403.04588/) (2024)
- [Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement](https://mlsanity.com/paper/2302.09318/) (2023)
- [Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RL](https://mlsanity.com/paper/2602.12087/) (2026)

## Abstract (arXiv)

In this work we explore the use of latent representations obtained from multiple input sensory modalities (such as images or sounds) in allowing an agent to learn and exploit policies over different subsets of input modalities. We propose a three-stage architecture that allows a reinforcement learning agent trained over a given sensory modality, to execute its task on a different sensory modality-for example, learning a visual policy over image inputs, and then execute such policy when only sound inputs are available. We show that the generalized policies achieve better out-of-the-box performance when compared to different baselines. Moreover, we show this holds in different OpenAI gym and video game environments, even when using different multimodal generative models and reinforcement learning algorithms.