# Embodied Multimodal Multitask Learning

arXiv: https://arxiv.org/abs/1902.01385 · HTML: https://mlsanity.com/paper/1902.01385/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2019.

## Built on by (most cited first, 13 of 13)

- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs](https://mlsanity.com/paper/2206.04674/) (2022)
- [Towards Versatile Embodied Navigation](https://mlsanity.com/paper/2210.16822/) (2022)
- [HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving](https://mlsanity.com/paper/2102.11344/) (2021)
- [MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning](https://mlsanity.com/paper/2209.01416/) (2022)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)
- [SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments](https://mlsanity.com/paper/2604.22409/) (2026)
- [Spot the Difference: A Novel Task for Embodied Agents in Changing Environments](https://mlsanity.com/paper/2204.08502/) (2022)
- [MIMONet: Multi-Input Multi-Output On-Device Deep Learning](https://mlsanity.com/paper/2307.11962/) (2023)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning](https://mlsanity.com/paper/2403.01483/) (2024)

## Abstract (arXiv)

Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual representations, and align them with each other. This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks. We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments. We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors.