# Random Forests versus Neural Networks - What's Best for Camera Localization?

arXiv: https://arxiv.org/abs/1609.05797 · HTML: https://mlsanity.com/paper/1609.05797/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2016.

## Built on by (most cited first, 15 of 15)

- [LaMAR: Benchmarking Localization and Mapping for Augmented Reality](https://mlsanity.com/paper/2210.10770/) (2022)
- [MeshLoc: Mesh-Based Visual Localization](https://mlsanity.com/paper/2207.10762/) (2022)
- [P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching](https://mlsanity.com/paper/2103.01055/) (2021)
- [Visual Camera Re-Localization Using Graph Neural Networks and Relative Pose Supervision](https://mlsanity.com/paper/2104.02538/) (2021)
- [Large-scale Localization Datasets in Crowded Indoor Spaces](https://mlsanity.com/paper/2105.08941/) (2021)
- [Investigating the Role of Image Retrieval for Visual Localization -- An exhaustive benchmark](https://mlsanity.com/paper/2205.15761/) (2022)
- [2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds](https://mlsanity.com/paper/2308.05667/) (2023)
- [HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization with Transformer](https://mlsanity.com/paper/2305.03595/) (2023)
- [How Privacy-Preserving are Line Clouds? Recovering Scene Details from 3D Lines](https://mlsanity.com/paper/2103.05086/) (2021)
- [Continual Learning for Image-Based Camera Localization](https://mlsanity.com/paper/2108.09112/) (2021)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [A Guide to Structureless Visual Localization](https://mlsanity.com/paper/2504.17636/) (2025)
- [ImPosing: Implicit Pose Encoding for Efficient Visual Localization](https://mlsanity.com/paper/2205.02638/) (2022)
- [Deep Weakly Supervised Positioning](https://mlsanity.com/paper/2104.04866/) (2021)
- [SGL: Structure Guidance Learning for Camera Localization](https://mlsanity.com/paper/2304.05571/) (2023)

## Abstract (arXiv)

This work addresses the task of camera localization in a known 3D scene given a single input RGB image. State-of-the-art approaches accomplish this in two steps: firstly, regressing for every pixel in the image its 3D scene coordinate and subsequently, using these coordinates to estimate the final 6D camera pose via RANSAC. To solve the first step, Random Forests (RFs) are typically used. On the other hand, Neural Networks (NNs) reign in many dense regression tasks, but are not test-time efficient. We ask the question: which of the two is best for camera localization? To address this, we make two method contributions: (1) a test-time efficient NN architecture which we term a ForestNet that is derived and initialized from a RF, and (2) a new fully-differentiable robust averaging technique for regression ensembles which can be trained end-to-end with a NN. Our experimental findings show that for scene coordinate regression, traditional NN architectures are superior to test-time efficient RFs and ForestNets, however, this does not translate to final 6D camera pose accuracy where RFs and ForestNets perform slightly better. To summarize, our best method, a ForestNet with a robust average, which has an equivalent fast and lightweight RF, improves over the state-of-the-art for camera localization on the 7-Scenes dataset. While this work focuses on scene coordinate regression for camera localization, our innovations may also be applied to other continuous regression tasks.