# Full-Frame Scene Coordinate Regression for Image-Based Localization

arXiv: https://arxiv.org/abs/1802.03237 · HTML: https://mlsanity.com/paper/1802.03237/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2018.

## Built on by (most cited first, 13 of 13)

- [Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization](https://mlsanity.com/paper/2412.08376/) (2024)
- [Visual Localization via Few-Shot Scene Region Classification](https://mlsanity.com/paper/2208.06933/) (2022)
- [2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds](https://mlsanity.com/paper/2308.05667/) (2023)
- [HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization with Transformer](https://mlsanity.com/paper/2305.03595/) (2023)
- [Reassessing the Limitations of CNN Methods for Camera Pose Regression](https://mlsanity.com/paper/2108.07260/) (2021)
- [Continual Learning for Image-Based Camera Localization](https://mlsanity.com/paper/2108.09112/) (2021)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [A Critical Analysis of Image-based Camera Pose Estimation Techniques](https://mlsanity.com/paper/2201.05816/) (2022)
- [Improved Scene Landmark Detection for Camera Localization](https://mlsanity.com/paper/2401.18083/) (2024)
- [D2S: Representing sparse descriptors and 3D coordinates for camera relocalization](https://mlsanity.com/paper/2307.15250/) (2023)
- [Reprojection Errors as Prompts for Efficient Scene Coordinate Regression](https://mlsanity.com/paper/2409.04178/) (2024)
- [3D Gaussian Splatting aided Localization for Large and Complex Indoor-Environments](https://mlsanity.com/paper/2502.13803/) (2025)
- [SGL: Structure Guidance Learning for Camera Localization](https://mlsanity.com/paper/2304.05571/) (2023)

## Abstract (arXiv)

Image-based localization, or camera relocalization, is a fundamental problem in computer vision and robotics, and it refers to estimating camera pose from an image. Recent state-of-the-art approaches use learning based methods, such as Random Forests (RFs) and Convolutional Neural Networks (CNNs), to regress for each pixel in the image its corresponding position in the scene's world coordinate frame, and solve the final pose via a RANSAC-based optimization scheme using the predicted correspondences. In this paper, instead of in a patch-based manner, we propose to perform the scene coordinate regression in a full-frame manner to make the computation efficient at test time and, more importantly, to add more global context to the regression process to improve the robustness. To do so, we adopt a fully convolutional encoder-decoder neural network architecture which accepts a whole image as input and produces scene coordinate predictions for all pixels in the image. However, using more global context is prone to overfitting. To alleviate this issue, we propose to use data augmentation to generate more data for training. In addition to the data augmentation in 2D image space, we also augment the data in 3D space. We evaluate our approach on the publicly available 7-Scenes dataset, and experiments show that it has better scene coordinate predictions and achieves state-of-the-art results in localization with improved robustness on the hardest frames (e.g., frames with repeated structures).