# Visual Semantic Navigation using Scene Priors

arXiv: https://arxiv.org/abs/1810.06543 · HTML: https://mlsanity.com/paper/1810.06543/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2018.

## Built on by (most cited first, 30 of 128)

- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning](https://mlsanity.com/paper/2201.10029/) (2022)
- [ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation](https://mlsanity.com/paper/2301.13166/) (2023)
- [SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation](https://mlsanity.com/paper/2410.08189/) (2024)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav](https://mlsanity.com/paper/2301.07302/) (2023)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNav](https://mlsanity.com/paper/2303.07798/) (2023)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [Memory-Augmented Reinforcement Learning for Image-Goal Navigation](https://mlsanity.com/paper/2101.05181/) (2021)
- [No RL, No Simulation: Learning to Navigate without Navigating](https://mlsanity.com/paper/2110.09470/) (2021)
- [PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators](https://mlsanity.com/paper/2406.20083/) (2024)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks](https://mlsanity.com/paper/2108.01176/) (2021)
- [VTNet: Visual Transformer Network for Object Goal Navigation](https://mlsanity.com/paper/2105.09447/) (2021)
- [Learning to Map for Active Semantic Goal Navigation](https://mlsanity.com/paper/2106.15648/) (2021)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022)
- [Visual Navigation with Spatial Attention](https://mlsanity.com/paper/2104.09807/) (2021)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [Continuous Scene Representations for Embodied AI](https://mlsanity.com/paper/2203.17251/) (2022)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Hierarchical Object-to-Zone Graph for Object Navigation](https://mlsanity.com/paper/2109.02066/) (2021)

## Abstract (arXiv)

How do humans navigate to target objects in novel scenes? Do we use the semantic/functional priors we have built over years to efficiently search and navigate? For example, to search for mugs, we search cabinets near the coffee machine and for fruits we try the fridge. In this work, we focus on incorporating semantic priors in the task of semantic navigation. We propose to use Graph Convolutional Networks for incorporating the prior knowledge into a deep reinforcement learning framework. The agent uses the features from the knowledge graph to predict the actions. For evaluation, we use the AI2-THOR framework. Our experiments show how semantic knowledge improves performance significantly. More importantly, we show improvement in generalization to unseen scenes and/or objects. The supplementary video can be accessed at the following link: https://youtu.be/otKjuO805dE .