# Active Semantic Localization with Graph Neural Embedding

arXiv: https://arxiv.org/abs/2305.06141 · HTML: https://mlsanity.com/paper/2305.06141/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [PlaNet - Photo Geolocation with Convolutional Neural Networks](https://mlsanity.com/paper/1602.05314/) (2016)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [On the Performance of ConvNet Features for Place Recognition](https://mlsanity.com/paper/1501.04158/) (2015)
- [Semantic Visual Localization](https://mlsanity.com/paper/1712.05773/) (2017)
- [Auxiliary Tasks Speed Up Learning PointGoal Navigation](https://mlsanity.com/paper/2007.04561/) (2020)
- [ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data](https://mlsanity.com/paper/2203.08321/) (2022)
- [Q-learning with Nearest Neighbors](https://mlsanity.com/paper/1802.03900/) (2018)
- [The Revisiting Problem in Simultaneous Localization and Mapping: A Survey on Visual Loop Closure Detection](https://mlsanity.com/paper/2204.12831/) (2022)
- [Robot Localization in Floor Plans Using a Room Layout Edge Extraction Network](https://mlsanity.com/paper/1903.01804/) (2019)
- [Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents](https://mlsanity.com/paper/2009.03231/) (2020)
- [Cross-Dataset Collaborative Learning for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2103.11351/) (2021)
- [iBoW-LCD: An Appearance-based Loop Closure Detection Approach using Incremental Bags of Binary Words](https://mlsanity.com/paper/1802.05909/) (2018)
- [ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Framework](https://mlsanity.com/paper/2204.01080/) (2022)
- [Active Neural Localization](https://mlsanity.com/paper/1801.08214/) (2018)
- [Learning Deep NBNN Representations for Robust Place Categorization](https://mlsanity.com/paper/1702.07898/) (2017)
- [Deep Active Localization](https://mlsanity.com/paper/1903.01669/) (2019)
- [Active Domain-Invariant Self-Localization Using Ego-Centric and World-Centric Maps](https://mlsanity.com/paper/2204.10497/) (2022)

## Built on by (most cited first, 1 of 1)

- [Introspection in Learned Semantic Scene Graph Localisation](https://mlsanity.com/paper/2510.07053/) (2025)

## Abstract (arXiv)

Semantic localization, i.e., robot self-localization with semantic image modality, is critical in recently emerging embodied AI applications (e.g., point-goal navigation, object-goal navigation, vision language navigation) and topological mapping applications (e.g., graph neural SLAM, ego-centric topological map). However, most existing works on semantic localization focus on passive vision tasks without viewpoint planning, or rely on additional rich modalities (e.g., depth measurements). Thus, the problem is largely unsolved. In this work, we explore a lightweight, entirely CPU-based, domain-adaptive semantic localization framework, called graph neural localizer. Our approach is inspired by two recently emerging technologies: (1) Scene graph, which combines the viewpoint- and appearance- invariance of local and global features; (2) Graph neural network, which enables direct learning/recognition of graph data (i.e., non-vector data). Specifically, a graph convolutional neural network is first trained as a scene graph classifier for passive vision, and then its knowledge is transferred to a reinforcement-learning planner for active vision. Experiments on two scenarios, self-supervised learning and unsupervised domain adaptation, using a photo-realistic Habitat simulator validate the effectiveness of the proposed method.