# Locking On: Leveraging Dynamic Vehicle-Imposed Motion Constraints to Improve Visual Localization

arXiv: https://arxiv.org/abs/2306.17529 · HTML: https://mlsanity.com/paper/2306.17529/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 73 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 30 of 34)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [Hierarchical Scene Coordinate Classification and Regression for Visual Localization](https://mlsanity.com/paper/1909.06216/) (2019)
- [YOLOP: You Only Look Once for Panoptic Driving Perception](https://mlsanity.com/paper/2108.11250/) (2021)
- [Robust Image Retrieval-based Visual Localization using Kapture](https://mlsanity.com/paper/2007.13867/) (2020)
- [VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry](https://mlsanity.com/paper/1804.08366/) (2018)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Semantic Visual Localization](https://mlsanity.com/paper/1712.05773/) (2017)
- [Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation](https://mlsanity.com/paper/1810.13049/) (2018)
- [X-View: Graph-Based Semantic Multi-View Localization](https://mlsanity.com/paper/1709.09905/) (2017)
- [Benchmarking Image Retrieval for Visual Localization](https://mlsanity.com/paper/2011.11946/) (2020)
- [Ford Multi-AV Seasonal Dataset](https://mlsanity.com/paper/2003.07969/) (2020)
- [LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics](https://mlsanity.com/paper/1804.05526/) (2018)
- [GN-Net: The Gauss-Newton Loss for Multi-Weather Relocalization](https://mlsanity.com/paper/1904.11932/) (2019)
- [Quaternion kinematics for the error-state Kalman filter](https://mlsanity.com/paper/1711.02508/) (2017)
- [Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization](https://mlsanity.com/paper/1809.01019/) (2018)
- [VS-Net: Voting with Segmentation for Visual Localization](https://mlsanity.com/paper/2105.10886/) (2021)
- [Fine-Grained Segmentation Networks: Self-Supervised Segmentation for Improved Long-Term Visual Localization](https://mlsanity.com/paper/1908.06387/) (2019)
- [Deep Multi-Task Learning for Joint Localization, Perception, and Prediction](https://mlsanity.com/paper/2101.06720/) (2021)
- [Localization Requirements for Autonomous Vehicles](https://mlsanity.com/paper/1906.01061/) (2019)
- [Long-term Visual Localization using Semantically Segmented Images](https://mlsanity.com/paper/1801.05269/) (2018)
- [Aerial Imagery based LIDAR Localization for Autonomous Vehicles](https://mlsanity.com/paper/2003.11192/) (2020)

## Abstract (arXiv)

Most 6-DoF localization and SLAM systems use static landmarks but ignore dynamic objects because they cannot be usefully incorporated into a typical pipeline. Where dynamic objects have been incorporated, typical approaches have attempted relatively sophisticated identification and localization of these objects, limiting their robustness or general utility. In this research, we propose a middle ground, demonstrated in the context of autonomous vehicles, using dynamic vehicles to provide limited pose constraint information in a 6-DoF frame-by-frame PnP-RANSAC localization pipeline. We refine initial pose estimates with a motion model and propose a method for calculating the predicted quality of future pose estimates, triggered based on whether or not the autonomous vehicle's motion is constrained by the relative frame-to-frame location of dynamic vehicles in the environment. Our approach detects and identifies suitable dynamic vehicles to define these pose constraints to modify a pose filter, resulting in improved recall across a range of localization tolerances from 0.25m to 5m, compared to a state-of-the-art baseline single image PnP method and its vanilla pose filtering. Our constraint detection system is active for approximately 35% of the time on the Ford AV dataset and localization is particularly improved when the constraint detection is active.