# Learning Matchable Image Transformations for Long-term Metric Visual Localization

arXiv: https://arxiv.org/abs/1904.01080 · HTML: https://mlsanity.com/paper/1904.01080/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2019.

## Built on by (most cited first, 8 of 8)

- [Where is your place, Visual Place Recognition?](https://mlsanity.com/paper/2103.06443/) (2021)
- [Visual Cross-View Metric Localization with Dense Uncertainty Estimates](https://mlsanity.com/paper/2208.08519/) (2022)
- [Learned Camera Gain and Exposure Control for Improved Visual Feature Detection and Matching](https://mlsanity.com/paper/2102.04341/) (2021)
- [Benchmarking Visual Localization for Autonomous Navigation](https://mlsanity.com/paper/2203.13048/) (2022)
- [Rover Relocalization for Mars Sample Return by Virtual Template Synthesis and Matching](https://mlsanity.com/paper/2103.03395/) (2021)
- [Multi-modal Visual Place Recognition in Dynamics-Invariant Perception Space](https://mlsanity.com/paper/2105.07800/) (2021)
- [Sparse Semantic Map-Based Monocular Localization in Traffic Scenes Using Learned 2D-3D Point-Line Correspondences](https://mlsanity.com/paper/2210.04543/) (2022)
- [Locking On: Leveraging Dynamic Vehicle-Imposed Motion Constraints to Improve Visual Localization](https://mlsanity.com/paper/2306.17529/) (2023)

## Abstract (arXiv)

Long-term metric self-localization is an essential capability of autonomous mobile robots, but remains challenging for vision-based systems due to appearance changes caused by lighting, weather, or seasonal variations. While experience-based mapping has proven to be an effective technique for bridging the `appearance gap,' the number of experiences required for reliable metric localization over days or months can be very large, and methods for reducing the necessary number of experiences are needed for this approach to scale. Taking inspiration from color constancy theory, we learn a nonlinear RGB-to-grayscale mapping that explicitly maximizes the number of inlier feature matches for images captured under different lighting and weather conditions, and use it as a pre-processing step in a conventional single-experience localization pipeline to improve its robustness to appearance change. We train this mapping by approximating the target non-differentiable localization pipeline with a deep neural network, and find that incorporating a learned low-dimensional context feature can further improve cross-appearance feature matching. Using synthetic and real-world datasets, we demonstrate substantial improvements in localization performance across day-night cycles, enabling continuous metric localization over a 30-hour period using a single mapping experience, and allowing experience-based localization to scale to long deployments with dramatically reduced data requirements.