# What Is Near?: Room Locality Learning for Enhanced Robot Vision-Language-Navigation in Indoor Living Environments

arXiv: https://arxiv.org/abs/2309.05036 · HTML: https://mlsanity.com/paper/2309.05036/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 37 references that are in the map.
- First posted on arXiv 10 Sep 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2202.11742/) (2022)
- [Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation](https://mlsanity.com/paper/1905.12255/) (2019)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [Visual Semantic Navigation using Scene Priors](https://mlsanity.com/paper/1810.06543/) (2018)
- [Topological Planning with Transformers for Vision-and-Language Navigation](https://mlsanity.com/paper/2012.05292/) (2020)
- [Occupancy Anticipation for Efficient Exploration and Navigation](https://mlsanity.com/paper/2008.09285/) (2020)
- [Cross-modal Map Learning for Vision and Language Navigation](https://mlsanity.com/paper/2203.05137/) (2022)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [Less is More: Generating Grounded Navigation Instructions from Landmarks](https://mlsanity.com/paper/2111.12872/) (2021)
- [Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views](https://mlsanity.com/paper/2010.01191/) (2020)
- [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022)
- [CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis](https://mlsanity.com/paper/1904.01920/) (2019)
- [Learning from Unlabeled 3D Environments for Vision-and-Language Navigation](https://mlsanity.com/paper/2208.11781/) (2022)
- [SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation](https://mlsanity.com/paper/2012.04512/) (2020)
- [Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation](https://mlsanity.com/paper/1911.07450/) (2019)

## Built on by (most cited first, 2 of 2)

- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation](https://mlsanity.com/paper/2409.05583/) (2024)

## Abstract (arXiv)

Humans use their knowledge of common house layouts obtained from previous experiences to predict nearby rooms while navigating in new environments. This greatly helps them navigate previously unseen environments and locate their target room. To provide layout prior knowledge to navigational agents based on common human living spaces, we propose WIN (textitWhat textitIs textitNear), a commonsense learning model for Vision Language Navigation (VLN) tasks. VLN requires an agent to traverse indoor environments based on descriptive navigational instructions. Unlike existing layout learning works, WIN predicts the local neighborhood map based on prior knowledge of living spaces and current observation, operating on an imagined global map of the entire environment. The model infers neighborhood regions based on visual cues of current observations, navigational history, and layout common sense. We show that local-global planning based on locality knowledge and predicting the indoor layout allows the agent to efficiently select the appropriate action. Specifically, we devised a cross-modal transformer that utilizes this locality prior for decision-making in addition to visual inputs and instructions. Experimental results show that locality learning using WIN provides better generalizability compared to classical VLN agents in unseen environments. Our model performs favorably on standard VLN metrics, with Success Rate 68% and Success weighted by Path Length 63% in unseen environments.