# Semantic Visual Localization

arXiv: https://arxiv.org/abs/1712.05773 · HTML: https://mlsanity.com/paper/1712.05773/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2017.

## Built on by (most cited first, 30 of 54)

- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [LaMAR: Benchmarking Localization and Mapping for Augmented Reality](https://mlsanity.com/paper/2210.10770/) (2022)
- [MeshLoc: Mesh-Based Visual Localization](https://mlsanity.com/paper/2207.10762/) (2022)
- [Is Geometry Enough for Matching in Visual Localization?](https://mlsanity.com/paper/2203.12979/) (2022)
- [GLACE: Global Local Accelerated Coordinate Encoding](https://mlsanity.com/paper/2406.04340/) (2024)
- [SOLD2: Self-supervised Occlusion-aware Line Description and Detection](https://mlsanity.com/paper/2104.03362/) (2021)
- [Extreme Rotation Estimation using Dense Correlation Volumes](https://mlsanity.com/paper/2104.13530/) (2021)
- [Text2Pos: Text-to-Point-Cloud Cross-Modal Localization](https://mlsanity.com/paper/2203.15125/) (2022)
- [SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding](https://mlsanity.com/paper/2306.05407/) (2023)
- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [SFD2: Semantic-guided Feature Detection and Description](https://mlsanity.com/paper/2304.14845/) (2023)
- [Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion](https://mlsanity.com/paper/2205.02830/) (2022)
- [The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement](https://mlsanity.com/paper/2404.10438/) (2024)
- [Visual Localization using Imperfect 3D Models from the Internet](https://mlsanity.com/paper/2304.05947/) (2023)
- [Segment Anything Model is a Good Teacher for Local Feature Learning](https://mlsanity.com/paper/2309.16992/) (2023)
- ["Where am I?" Scene Retrieval with Language](https://mlsanity.com/paper/2404.14565/) (2024)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Place recognition survey: An update on deep learning approaches](https://mlsanity.com/paper/2106.10458/) (2021)
- [CrowdDriven: A New Challenging Dataset for Outdoor Visual Localization](https://mlsanity.com/paper/2109.04527/) (2021)
- [BEV-Locator: An End-to-end Visual Semantic Localization Network Using Multi-View Images](https://mlsanity.com/paper/2211.14927/) (2022)
- [Learning to Produce Semi-dense Correspondences for Visual Localization](https://mlsanity.com/paper/2402.08359/) (2024)
- [CGiS-Net: Aggregating Colour, Geometry and Implicit Semantic Features for Indoor Place Recognition](https://mlsanity.com/paper/2202.02070/) (2022)
- [Searching from Area to Point: A Hierarchical Framework for Semantic-Geometric Combined Feature Matching](https://mlsanity.com/paper/2305.00194/) (2023)
- [PSE-Match: A Viewpoint-free Place Recognition Method with Parallel Semantic Embedding](https://mlsanity.com/paper/2108.00552/) (2021)
- [Unified Representation of Geometric Primitives for Graph-SLAM Optimization Using Decomposed Quadrics](https://mlsanity.com/paper/2108.08957/) (2021)
- [Anomaly Detection of Defect using Energy of Point Pattern Features within Random Finite Set Framework](https://mlsanity.com/paper/2108.12159/) (2021)
- [S3LAM: Structured Scene SLAM](https://mlsanity.com/paper/2109.07339/) (2021)
- [Learning-based Relational Object Matching Across Views](https://mlsanity.com/paper/2305.02398/) (2023)
- [S-BEV: Semantic Birds-Eye View Representation for Weather and Lighting Invariant 3-DoF Localization](https://mlsanity.com/paper/2101.09569/) (2021)

## Abstract (arXiv)

Robust visual localization under a wide range of viewing conditions is a fundamental problem in computer vision. Handling the difficult cases of this problem is not only very challenging but also of high practical relevance, e.g., in the context of life-long localization for augmented reality or autonomous robots. In this paper, we propose a novel approach based on a joint 3D geometric and semantic understanding of the world, enabling it to succeed under conditions where previous approaches failed. Our method leverages a novel generative model for descriptor learning, trained on semantic scene completion as an auxiliary task. The resulting 3D descriptors are robust to missing observations by encoding high-level 3D geometric and semantic information. Experiments on several challenging large-scale localization datasets demonstrate reliable localization under extreme viewpoint, illumination, and geometry changes.