# Multimodal Attention Networks for Low-Level Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/1911.12377 · HTML: https://mlsanity.com/paper/1911.12377/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2019.

## Built on by (most cited first, 17 of 17)

- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [Improving Vision-and-Language Navigation by Generating Future-View Image Semantics](https://mlsanity.com/paper/2304.04907/) (2023)
- [CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations](https://mlsanity.com/paper/2207.02185/) (2022)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Out of the Box: Embodied Navigation in the Real World](https://mlsanity.com/paper/2105.05873/) (2021)
- [Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation](https://mlsanity.com/paper/2503.18065/) (2025)
- [Active Test-time Vision-Language Navigation](https://mlsanity.com/paper/2506.06630/) (2025)
- [Embodied Agents for Efficient Exploration and Smart Scene Description](https://mlsanity.com/paper/2301.07150/) (2023)
- [Vision-Language Navigation with Continual Learning](https://mlsanity.com/paper/2409.02561/) (2024)
- [Spot the Difference: A Novel Task for Embodied Agents in Changing Environments](https://mlsanity.com/paper/2204.08502/) (2022)
- [Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigation](https://mlsanity.com/paper/2504.08806/) (2025)
- [Find a Way Forward: a Language-Guided Semantic Map Navigator](https://mlsanity.com/paper/2203.03183/) (2022)
- [Embodied Navigation at the Art Gallery](https://mlsanity.com/paper/2204.09069/) (2022)
- [What Is Near?: Room Locality Learning for Enhanced Robot Vision-Language-Navigation in Indoor Living Environments](https://mlsanity.com/paper/2309.05036/) (2023)
- [AIGeN: An Adversarial Approach for Instruction Generation in VLN](https://mlsanity.com/paper/2404.10054/) (2024)
- [Vision-and-Language Navigation Generative Pretrained Transformer](https://mlsanity.com/paper/2405.16994/) (2024)

## Abstract (arXiv)

Vision-and-Language Navigation (VLN) is a challenging task in which an agent needs to follow a language-specified path to reach a target destination. The goal gets even harder as the actions available to the agent get simpler and move towards low-level, atomic interactions with the environment. This setting takes the name of low-level VLN. In this paper, we strive for the creation of an agent able to tackle three key issues: multi-modality, long-term dependencies, and adaptability towards different locomotive settings. To that end, we devise "Perceive, Transform, and Act" (PTA): a fully-attentive VLN architecture that leaves the recurrent approach behind and the first Transformer-like architecture incorporating three different modalities - natural language, images, and low-level actions for the agent control. In particular, we adopt an early fusion strategy to merge lingual and visual information efficiently in our encoder. We then propose to refine the decoding phase with a late fusion extension between the agent's history of actions and the perceptual modalities. We experimentally validate our model on two datasets: PTA achieves promising results in low-level VLN on R2R and achieves good performance in the recently proposed R4R benchmark. Our code is publicly available at https://github.com/aimagelab/perceive-transform-and-act.