# Hierarchical Scene Coordinate Classification and Regression for Visual Localization

arXiv: https://arxiv.org/abs/1909.06216 · HTML: https://mlsanity.com/paper/1909.06216/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2019.

## Built on by (most cited first, 30 of 93)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [RoMa: Robust Dense Feature Matching](https://mlsanity.com/paper/2305.15404/) (2023)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Map-free Visual Relocalization: Metric Pose Relative to a Single Image](https://mlsanity.com/paper/2210.05494/) (2022)
- [Accelerated Coordinate Encoding: Learning to Relocalize in Minutes using RGB and Poses](https://mlsanity.com/paper/2305.14059/) (2023)
- [Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization](https://mlsanity.com/paper/2412.08376/) (2024)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [Is Geometry Enough for Matching in Visual Localization?](https://mlsanity.com/paper/2203.12979/) (2022)
- [GLACE: Global Local Accelerated Coordinate Encoding](https://mlsanity.com/paper/2406.04340/) (2024)
- [SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Reality](https://mlsanity.com/paper/2409.14067/) (2024)
- [P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching](https://mlsanity.com/paper/2103.01055/) (2021)
- [Map-Relative Pose Regression for Visual Re-Localization](https://mlsanity.com/paper/2404.09884/) (2024)
- [Visual Localization via Few-Shot Scene Region Classification](https://mlsanity.com/paper/2208.06933/) (2022)
- [The NeRFect Match: Exploring NeRF Features for Visual Localization](https://mlsanity.com/paper/2403.09577/) (2024)
- [Large-scale Localization Datasets in Crowded Indoor Spaces](https://mlsanity.com/paper/2105.08941/) (2021)
- [VS-Net: Voting with Segmentation for Visual Localization](https://mlsanity.com/paper/2105.10886/) (2021)
- [NeRF-Loc: Visual Localization with Conditional Neural Radiance Field](https://mlsanity.com/paper/2304.07979/) (2023)
- [Learning Camera Localization via Dense Scene Matching](https://mlsanity.com/paper/2103.16792/) (2021)
- [CrossLoc: Scalable Aerial Localization Assisted by Multimodal Synthetic Data](https://mlsanity.com/paper/2112.09081/) (2021)
- [Investigating the Role of Image Retrieval for Visual Localization -- An exhaustive benchmark](https://mlsanity.com/paper/2205.15761/) (2022)
- [2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds](https://mlsanity.com/paper/2308.05667/) (2023)
- [NeuMap: Neural Coordinate Mapping by Auto-Transdecoder for Camera Localization](https://mlsanity.com/paper/2211.11177/) (2022)
- [HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization with Transformer](https://mlsanity.com/paper/2305.03595/) (2023)
- [Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences](https://mlsanity.com/paper/2404.06337/) (2024)
- [PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields](https://mlsanity.com/paper/2312.10649/) (2023)
- [The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement](https://mlsanity.com/paper/2404.10438/) (2024)
- [Reassessing the Limitations of CNN Methods for Camera Pose Regression](https://mlsanity.com/paper/2108.07260/) (2021)
- [Continual Learning for Image-Based Camera Localization](https://mlsanity.com/paper/2108.09112/) (2021)
- [DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching](https://mlsanity.com/paper/2306.12547/) (2023)

## Abstract (arXiv)

Visual localization is critical to many applications in computer vision and robotics. To address single-image RGB localization, state-of-the-art feature-based methods match local descriptors between a query image and a pre-built 3D model. Recently, deep neural networks have been exploited to regress the mapping between raw pixels and 3D coordinates in the scene, and thus the matching is implicitly performed by the forward pass through the network. However, in a large and ambiguous environment, learning such a regression task directly can be difficult for a single network. In this work, we present a new hierarchical scene coordinate network to predict pixel scene coordinates in a coarse-to-fine manner from a single RGB image. The network consists of a series of output layers, each of them conditioned on the previous ones. The final output layer predicts the 3D coordinates and the others produce progressively finer discrete location labels. The proposed method outperforms the baseline regression-only network and allows us to train compact models which scale robustly to large environments. It sets a new state-of-the-art for single-image RGB localization performance on the 7-Scenes, 12-Scenes, Cambridge Landmarks datasets, and three combined scenes. Moreover, for large-scale outdoor localization on the Aachen Day-Night dataset, we present a hybrid approach which outperforms existing scene coordinate regression methods, and reduces significantly the performance gap w.r.t. explicit feature matching methods.