# Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression

arXiv: https://arxiv.org/abs/2403.10297 · HTML: https://mlsanity.com/paper/2403.10297/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [DISK: Learning local features with policy gradient](https://mlsanity.com/paper/2006.13566/) (2020)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC](https://mlsanity.com/paper/2002.12324/) (2020)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [Hierarchical Scene Coordinate Classification and Regression for Visual Localization](https://mlsanity.com/paper/1909.06216/) (2019)
- [AtLoc: Attention Guided Camera Localization](https://mlsanity.com/paper/1909.03557/) (2019)
- [Revealing Scenes by Inverting Structure from Motion Reconstructions](https://mlsanity.com/paper/1904.03303/) (2019)
- [Learning to Navigate the Energy Landscape](https://mlsanity.com/paper/1603.05772/) (2016)
- [To Learn or Not to Learn: Visual Localization from Essential Matrices](https://mlsanity.com/paper/1908.01293/) (2019)
- [KFNet: Learning Temporal Camera Relocalization using Kalman Filtering](https://mlsanity.com/paper/2003.10629/) (2020)
- [Visual Localization via Few-Shot Scene Region Classification](https://mlsanity.com/paper/2208.06933/) (2022)
- [Reassessing the Limitations of CNN Methods for Camera Pose Regression](https://mlsanity.com/paper/2108.07260/) (2021)
- [Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization](https://mlsanity.com/paper/2310.06984/) (2023)
- [PoseGAN: A Pose-to-Image Translation Framework for Camera Localization](https://mlsanity.com/paper/2006.12712/) (2020)
- [RenderNet: Visual Relocalization Using Virtual Viewpoints in Large-Scale Indoor Environments](https://mlsanity.com/paper/2207.12579/) (2022)

## Built on by (most cited first, 2 of 2)

- [Improved 3D Point-Line Mapping Regression for Camera Relocalization](https://mlsanity.com/paper/2502.20814/) (2025)
- [A-SCoRe: Attention-based Scene Coordinate Regression for wide-ranging scenarios](https://mlsanity.com/paper/2503.13982/) (2025)

## Abstract (arXiv)

Classical structural-based visual localization methods offer high accuracy but face trade-offs in terms of storage, speed, and privacy. A recent innovation, keypoint scene coordinate regression (KSCR) named D2S addresses these issues by leveraging graph attention networks to enhance keypoint relationships and predict their 3D coordinates using a simple multilayer perceptron (MLP). Camera pose is then determined via PnP+RANSAC, using established 2D-3D correspondences. While KSCR achieves competitive results, rivaling state-of-the-art image-retrieval methods like HLoc across multiple benchmarks, its performance is hindered when data samples are limited due to the deep learning model's reliance on extensive data. This paper proposes a solution to this challenge by introducing a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF). By generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR's generalization capabilities in data-scarce environments. The proposed system could significantly improve localization accuracy by up to 50% and cost only a fraction of time for data synthesis. Furthermore, its modular design allows for the integration of multiple NeRFs, offering a versatile and efficient solution for visual localization. The implementation is publicly available at: https://github.com/ais-lab/DescriptorSynthesis4Feat2Map.