# HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization with Transformer

arXiv: https://arxiv.org/abs/2305.03595 · HTML: https://mlsanity.com/paper/2305.03595/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 92 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [DISK: Learning local features with policy gradient](https://mlsanity.com/paper/2006.13566/) (2020)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC](https://mlsanity.com/paper/2002.12324/) (2020)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [Modelling Uncertainty in Deep Learning for Camera Relocalization](https://mlsanity.com/paper/1509.05909/) (2015)
- [Patch2Pix: Epipolar-Guided Pixel-Level Correspondences](https://mlsanity.com/paper/2012.01909/) (2020)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [Learning to Compare Image Patches via Convolutional Neural Networks](https://mlsanity.com/paper/1504.03641/) (2015)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [Learning Feature Descriptors using Camera Pose Supervision](https://mlsanity.com/paper/2004.13324/) (2020)
- [Hierarchical Scene Coordinate Classification and Regression for Visual Localization](https://mlsanity.com/paper/1909.06216/) (2019)

## Built on by (most cited first, 26 of 26)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization](https://mlsanity.com/paper/2412.08376/) (2024)
- [GLACE: Global Local Accelerated Coordinate Encoding](https://mlsanity.com/paper/2406.04340/) (2024)
- [Compact 3D Gaussian Splatting For Dense Visual SLAM](https://mlsanity.com/paper/2403.11247/) (2024)
- [R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localization](https://mlsanity.com/paper/2501.01421/) (2025)
- [Differentiable Product Quantization for Memory Efficient Camera Relocalization](https://mlsanity.com/paper/2407.15540/) (2024)
- [A Guide to Structureless Visual Localization](https://mlsanity.com/paper/2504.17636/) (2025)
- [ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training](https://mlsanity.com/paper/2510.11605/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [D2S: Representing sparse descriptors and 3D coordinates for camera relocalization](https://mlsanity.com/paper/2307.15250/) (2023)
- [3D Gaussian Splatting aided Localization for Large and Complex Indoor-Environments](https://mlsanity.com/paper/2502.13803/) (2025)
- [A2-GNN: Angle-Annular GNN for Visual Descriptor-free Camera Relocalization](https://mlsanity.com/paper/2502.20036/) (2025)
- [Marginalized Bundle Adjustment: Multi-View Camera Pose from Monocular Depth Estimates](https://mlsanity.com/paper/2602.18906/) (2026)
- [NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired People](https://mlsanity.com/paper/2410.18109/) (2024)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Scene Coordinate Reconstruction Priors](https://mlsanity.com/paper/2510.12387/) (2025)
- [MACE: Mixture-of-Experts Accelerated Coordinate Encoding for Large-Scale Scene Localization and Rendering](https://mlsanity.com/paper/2510.14251/) (2025)
- [ImLoc: Revisiting Visual Localization with Image-based Representation](https://mlsanity.com/paper/2601.04185/) (2026)
- [Cross-Modal Visual Relocalization in Prior LiDAR Maps Utilizing Intensity Textures](https://mlsanity.com/paper/2412.01299/) (2024)
- [Improved 3D Point-Line Mapping Regression for Camera Relocalization](https://mlsanity.com/paper/2502.20814/) (2025)
- [ACE-SLAM: Scene Coordinate Regression for Neural Implicit Real-Time SLAM](https://mlsanity.com/paper/2512.14032/) (2025)
- [Robust Scene Coordinate Regression via Geometrically-Consistent Global Descriptors](https://mlsanity.com/paper/2512.17226/) (2025)
- [MultiLoc: Look Around As You Localize for Fast And Robust Visual Re-localization](https://mlsanity.com/paper/2603.27170/) (2026)
- [GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training](https://mlsanity.com/paper/2607.02486/) (2026)
- [LightLoc++: Sensor-Robust Representation Learning for Efficient Outdoor LiDAR Localization](https://mlsanity.com/paper/2608.15317/) (2026)
- [AutoCompass: Accurate Visual Localization on Public Maps by Learning from Weak Labels](https://mlsanity.com/paper/2609.02798/) (2026)

## Abstract (arXiv)

Visual localization is critical to many applications in computer vision and robotics. To address single-image RGB localization, state-of-the-art feature-based methods match local descriptors between a query image and a pre-built 3D model. Recently, deep neural networks have been exploited to regress the mapping between raw pixels and 3D coordinates in the scene, and thus the matching is implicitly performed by the forward pass through the network. However, in a large and ambiguous environment, learning such a regression task directly can be difficult for a single network. In this work, we present a new hierarchical scene coordinate network to predict pixel scene coordinates in a coarse-to-fine manner from a single RGB image. The proposed method, which is an extension of HSCNet, allows us to train compact models which scale robustly to large environments. It sets a new state-of-the-art for single-image localization on the 7-Scenes, 12 Scenes, Cambridge Landmarks datasets, and the combined indoor scenes.