# Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization

arXiv: https://arxiv.org/abs/1812.03402 · HTML: https://mlsanity.com/paper/1812.03402/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2018.

## Built on by (most cited first, 5 of 5)

- [Deep Visual Geo-localization Benchmark](https://mlsanity.com/paper/2204.03444/) (2022)
- [Where in the World is this Image? Transformer-based Geo-localization in the Wild](https://mlsanity.com/paper/2204.13861/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [PIGEON: Predicting Image Geolocations](https://mlsanity.com/paper/2307.05845/) (2023)
- [OpenStreetView-5M: The Many Roads to Global Visual Geolocation](https://mlsanity.com/paper/2404.18873/) (2024)

## Abstract (arXiv)

We present an approach that combines appearance and semantic information for 2D image-based localization (2D-VL) across large perceptual changes and time lags. Compared to appearance features, the semantic layout of a scene is generally more invariant to appearance variations. We use this intuition and propose a novel end-to-end deep attention-based framework that utilizes multimodal cues to generate robust embeddings for 2D-VL. The proposed attention module predicts a shared channel attention and modality-specific spatial attentions to guide the embeddings to focus on more reliable image regions. We evaluate our model against state-of-the-art (SOTA) methods on three challenging localization datasets. We report an average (absolute) improvement of 19% over current SOTA for 2D-VL. Furthermore, we present an extensive study demonstrating the contribution of each component of our model, showing 8--15% and 4% improvement from adding semantic information and our proposed attention module. We finally show the predicted attention maps to offer useful insights into our model.