# SC-wLS: Towards Interpretable Feed-forward Camera Re-localization

arXiv: https://arxiv.org/abs/2210.12748 · HTML: https://mlsanity.com/paper/2210.12748/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC](https://mlsanity.com/paper/2002.12324/) (2020)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [AtLoc: Attention Guided Camera Localization](https://mlsanity.com/paper/1909.03557/) (2019)
- [Deep Declarative Networks: A New Hope](https://mlsanity.com/paper/1909.04866/) (2019)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [KFNet: Learning Temporal Camera Relocalization using Kalman Filtering](https://mlsanity.com/paper/2003.10629/) (2020)
- [Physics Inspired Optimization on Semantic Transfer Features: An Alternative Method for Room Layout Estimation](https://mlsanity.com/paper/1707.00383/) (2017)
- [Local Supports Global: Deep Camera Relocalization with Sequence Enhancement](https://mlsanity.com/paper/1908.04391/) (2019)
- [Generalizing to the Open World: Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2103.15279/) (2021)
- [Smart Hypothesis Generation for Efficient and Robust Room Layout Estimation](https://mlsanity.com/paper/1910.12257/) (2019)
- [Exploiting Points and Lines in Regression Forests for RGB-D Camera Relocalization](https://mlsanity.com/paper/1710.10519/) (2017)
- [Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems](https://mlsanity.com/paper/2004.07931/) (2020)

## Built on by (most cited first, 14 of 14)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow](https://mlsanity.com/paper/2211.14020/) (2022)
- [NeuMap: Neural Coordinate Mapping by Auto-Transdecoder for Camera Localization](https://mlsanity.com/paper/2211.11177/) (2022)
- [DQS3D: Densely-matched Quantization-aware Semi-supervised 3D Detection](https://mlsanity.com/paper/2304.13031/) (2023)
- [FastMAC: Stochastic Spectral Sampling of Correspondence Graph](https://mlsanity.com/paper/2403.08770/) (2024)
- [SACReg: Scene-Agnostic Coordinate Regression for Visual Localization](https://mlsanity.com/paper/2307.11702/) (2023)
- [Improving Feature-based Visual Localization by Geometry-Aided Matching](https://mlsanity.com/paper/2211.08712/) (2022)
- [Marginalized Bundle Adjustment: Multi-View Camera Pose from Monocular Depth Estimates](https://mlsanity.com/paper/2602.18906/) (2026)
- [Activating Self-Attention for Multi-Scene Absolute Pose Regression](https://mlsanity.com/paper/2411.01443/) (2024)
- [VRS-NeRF: Visual Relocalization with Sparse Neural Radiance Field](https://mlsanity.com/paper/2404.09271/) (2024)
- [FaVoR: Features via Voxel Rendering for Camera Relocalization](https://mlsanity.com/paper/2409.07571/) (2024)
- [Dual-frame Fluid Motion Estimation with Test-time Optimization and Zero-divergence Loss](https://mlsanity.com/paper/2410.11934/) (2024)
- [Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences](https://mlsanity.com/paper/2211.07459/) (2022)
- [PRAM: Place Recognition Anywhere Model for Efficient Visual Localization](https://mlsanity.com/paper/2404.07785/) (2024)

## Abstract (arXiv)

Visual re-localization aims to recover camera poses in a known environment, which is vital for applications like robotics or augmented reality. Feed-forward absolute camera pose regression methods directly output poses by a network, but suffer from low accuracy. Meanwhile, scene coordinate based methods are accurate, but need iterative RANSAC post-processing, which brings challenges to efficient end-to-end training and inference. In order to have the best of both worlds, we propose a feed-forward method termed SC-wLS that exploits all scene coordinate estimates for weighted least squares pose regression. This differentiable formulation exploits a weight network imposed on 2D-3D correspondences, and requires pose supervision only. Qualitative results demonstrate the interpretability of learned weights. Evaluations on 7Scenes and Cambridge datasets show significantly promoted performance when compared with former feed-forward counterparts. Moreover, our SC-wLS method enables a new capability: self-supervised test-time adaptation on the weight network. Codes and models are publicly available.