# Poses as Queries: Image-to-LiDAR Map Localization with Transformers

arXiv: https://arxiv.org/abs/2305.04298 · HTML: https://mlsanity.com/paper/2305.04298/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 7 May 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Scan Context++: Structural Place Recognition Robust to Rotation and Lateral Variations in Urban Environments](https://mlsanity.com/paper/2109.13494/) (2021)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [DeepI2P: Image-to-Point Cloud Registration via Deep Classification](https://mlsanity.com/paper/2104.03501/) (2021)
- [CMRNet: Camera to LiDAR-Map Registration](https://mlsanity.com/paper/1906.10109/) (2019)
- [BVMatch: Lidar-based Place Recognition Using Bird's-eye View Images](https://mlsanity.com/paper/2109.00317/) (2021)
- [Monocular Camera Localization in Prior LiDAR Maps with 2D-3D Line Correspondences](https://mlsanity.com/paper/2004.00740/) (2020)
- [OpenCalib: A Multi-sensor Calibration Toolbox for Autonomous Driving](https://mlsanity.com/paper/2205.14087/) (2022)
- [CMRNet++: Map and Camera Agnostic Monocular Visual Localization in LiDAR Maps](https://mlsanity.com/paper/2004.13795/) (2020)
- [Monocular Vehicle Self-localization method based on Compact Semantic Map](https://mlsanity.com/paper/1805.06155/) (2018)

## Built on by (most cited first, 2 of 2)

- [An Overview about Emerging Technologies of Autonomous Driving](https://mlsanity.com/paper/2306.13302/) (2023)
- [VioLA: Aligning Videos to 2D LiDAR Scans](https://mlsanity.com/paper/2311.04783/) (2023)

## Abstract (arXiv)

High-precision vehicle localization with commercial setups is a crucial technique for high-level autonomous driving tasks. Localization with a monocular camera in LiDAR map is a newly emerged approach that achieves promising balance between cost and accuracy, but estimating pose by finding correspondences between such cross-modal sensor data is challenging, thereby damaging the localization accuracy. In this paper, we address the problem by proposing a novel Transformer-based neural network to register 2D images into 3D LiDAR map in an end-to-end manner. Poses are implicitly represented as high-dimensional feature vectors called pose queries and can be iteratively updated by interacting with the retrieved relevant information from cross-model features using attention mechanism in a proposed POse Estimator Transformer (POET) module. Moreover, we apply a multiple hypotheses aggregation method that estimates the final poses by performing parallel optimization on multiple randomly initialized pose queries to reduce the network uncertainty. Comprehensive analysis and experimental results on public benchmark conclude that the proposed image-to-LiDAR map localization network could achieve state-of-the-art performances in challenging cross-modal localization tasks.