# Continual Learning for Image-Based Camera Localization

arXiv: https://arxiv.org/abs/2108.09112 · HTML: https://mlsanity.com/paper/2108.09112/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 72 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 30 of 50)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC](https://mlsanity.com/paper/2002.12324/) (2020)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Using Hindsight to Anchor Past Knowledge in Continual Learning](https://mlsanity.com/paper/2002.08165/) (2020)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [Modelling Uncertainty in Deep Learning for Camera Relocalization](https://mlsanity.com/paper/1509.05909/) (2015)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Memory-Efficient Incremental Learning Through Feature Adaptation](https://mlsanity.com/paper/2004.00713/) (2020)
- [Continual Learning in Low-rank Orthogonal Subspaces](https://mlsanity.com/paper/2010.11635/) (2020)
- [Hierarchical Scene Coordinate Classification and Regression for Visual Localization](https://mlsanity.com/paper/1909.06216/) (2019)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)

## Built on by (most cited first, 18 of 18)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [LaMAR: Benchmarking Localization and Mapping for Augmented Reality](https://mlsanity.com/paper/2210.10770/) (2022)
- [Visual Localization via Few-Shot Scene Region Classification](https://mlsanity.com/paper/2208.06933/) (2022)
- [HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization with Transformer](https://mlsanity.com/paper/2305.03595/) (2023)
- [DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching](https://mlsanity.com/paper/2306.12547/) (2023)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [LiftFeat: 3D Geometry-Aware Local Feature Matching](https://mlsanity.com/paper/2505.03422/) (2025)
- [Differentiable Product Quantization for Memory Efficient Camera Relocalization](https://mlsanity.com/paper/2407.15540/) (2024)
- [Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval](https://mlsanity.com/paper/2201.09724/) (2022)
- [Federated Continual Graph Learning](https://mlsanity.com/paper/2411.18919/) (2024)
- [Improved Scene Landmark Detection for Camera Localization](https://mlsanity.com/paper/2401.18083/) (2024)
- [OFVL-MS: Once for Visual Localization across Multiple Indoor Scenes](https://mlsanity.com/paper/2308.11928/) (2023)
- [A2-GNN: Angle-Annular GNN for Visual Descriptor-free Camera Relocalization](https://mlsanity.com/paper/2502.20036/) (2025)
- [Progressive Conservative Adaptation for Evolving Target Domains](https://mlsanity.com/paper/2402.04573/) (2024)
- [Towards Global Localization using Multi-Modal Object-Instance Re-Identification](https://mlsanity.com/paper/2409.12002/) (2024)
- [Continual Hand-Eye Calibration for Open-world Robotic Manipulation](https://mlsanity.com/paper/2604.15814/) (2026)
- [Magnetic Indoor Localization through CNN Regression and Rotation Invariance](https://mlsanity.com/paper/2604.22896/) (2026)
- [GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training](https://mlsanity.com/paper/2607.02486/) (2026)

## Abstract (arXiv)

For several emerging technologies such as augmented reality, autonomous driving and robotics, visual localization is a critical component. Directly regressing camera pose/3D scene coordinates from the input image using deep neural networks has shown great potential. However, such methods assume a stationary data distribution with all scenes simultaneously available during training. In this paper, we approach the problem of visual localization in a continual learning setup -- whereby the model is trained on scenes in an incremental manner. Our results show that similar to the classification domain, non-stationary data induces catastrophic forgetting in deep networks for visual localization. To address this issue, a strong baseline based on storing and replaying images from a fixed buffer is proposed. Furthermore, we propose a new sampling method based on coverage score (Buff-CS) that adapts the existing sampling strategies in the buffering process to the problem of visual localization. Results demonstrate consistent improvements over standard buffering methods on two challenging datasets -- 7Scenes, 12Scenes, and also 19Scenes by combining the former scenes.