# MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation

arXiv: https://arxiv.org/abs/2103.11374 · HTML: https://mlsanity.com/paper/2103.11374/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 21 Mar 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Visual Semantic Navigation using Scene Priors](https://mlsanity.com/paper/1810.06543/) (2018)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [Episodic Curiosity through Reachability](https://mlsanity.com/paper/1810.02274/) (2018)
- [Occupancy Anticipation for Efficient Exploration and Navigation](https://mlsanity.com/paper/2008.09285/) (2020)
- [Neural Map: Structured Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.08360/) (2017)
- [MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments](https://mlsanity.com/paper/1712.03931/) (2017)
- [Control of Memory, Active Perception, and Action in Minecraft](https://mlsanity.com/paper/1605.09128/) (2016)
- [Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views](https://mlsanity.com/paper/2010.01191/) (2020)

## Built on by (most cited first, 11 of 11)

- [SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2108.11945/) (2021)
- [Learning Active Camera for Multi-Object Navigation](https://mlsanity.com/paper/2210.07505/) (2022)
- [GraphMapper: Efficient Visual Navigation by Scene Graph Generation](https://mlsanity.com/paper/2205.08325/) (2022)
- [Feudal Networks for Visual Navigation](https://mlsanity.com/paper/2402.12498/) (2024)
- [Find a Way Forward: a Language-Guided Semantic Map Navigator](https://mlsanity.com/paper/2203.03183/) (2022)
- [STRNet: Visual Navigation with Spatio-Temporal Representation through Dynamic Graph Aggregation](https://mlsanity.com/paper/2604.02829/) (2026)
- [Autonomy and Perception for Space Mining](https://mlsanity.com/paper/2109.12109/) (2021)
- [Task-Driven Graph Attention for Hierarchical Relational Object Navigation](https://mlsanity.com/paper/2306.13760/) (2023)
- [VSGM -- Enhance robot task understanding ability through visual semantic graph](https://mlsanity.com/paper/2105.08959/) (2021)
- [YOPO-Nav: Visual Navigation using 3DGS Graphs from One-Pass Videos](https://mlsanity.com/paper/2512.09903/) (2025)
- [FeudalNav: A Simple Framework for Visual Navigation](https://mlsanity.com/paper/2602.06974/) (2026)

## Abstract (arXiv)

Visual navigation for autonomous agents is a core task in the fields of computer vision and robotics. Learning-based methods, such as deep reinforcement learning, have the potential to outperform the classical solutions developed for this task; however, they come at a significantly increased computational load. Through this work, we design a novel approach that focuses on performing better or comparable to the existing learning-based solutions but under a clear time/computational budget. To this end, we propose a method to encode vital scene semantics such as traversable paths, unexplored areas, and observed scene objects -- alongside raw visual streams such as RGB, depth, and semantic segmentation masks -- into a semantically informed, top-down egocentric map representation. Further, to enable the effective use of this information, we introduce a novel 2-D map attention mechanism, based on the successful multi-layer Transformer networks. We conduct experiments on 3-D reconstructed indoor PointGoal visual navigation and demonstrate the effectiveness of our approach. We show that by using our novel attention schema and auxiliary rewards to better utilize scene semantics, we outperform multiple baselines trained with only raw inputs or implicit semantic information while operating with an 80% decrease in the agent's experience.