# Attention-Enhanced Cross-modal Localization Between 360 Images and Point Clouds

arXiv: https://arxiv.org/abs/2212.02757 · HTML: https://mlsanity.com/paper/2212.02757/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018)
- [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019)
- [AdaFusion: Visual-LiDAR Fusion with Adaptive Weights for Place Recognition](https://mlsanity.com/paper/2111.11739/) (2021)
- [Global visual localization in LiDAR-maps through shared 2D-3D embedding space](https://mlsanity.com/paper/1910.04871/) (2019)
- [i3dLoc: Image-to-range Cross-domain Localization Robust to Inconsistent Environmental Conditions](https://mlsanity.com/paper/2105.12883/) (2021)

## Built on by (most cited first, 1 of 1)

- [LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localization](https://mlsanity.com/paper/2312.16648/) (2023)

## Abstract (arXiv)

Visual localization plays an important role for intelligent robots and autonomous driving, especially when the accuracy of GNSS is unreliable. Recently, camera localization in LiDAR maps has attracted more and more attention for its low cost and potential robustness to illumination and weather changes. However, the commonly used pinhole camera has a narrow Field-of-View, thus leading to limited information compared with the omni-directional LiDAR data. To overcome this limitation, we focus on correlating the information of 360 equirectangular images to point clouds, proposing an end-to-end learnable network to conduct cross-modal visual localization by establishing similarity in high-dimensional feature space. Inspired by the attention mechanism, we optimize the network to capture the salient feature for comparing images and point clouds. We construct several sequences containing 360 equirectangular images and corresponding point clouds based on the KITTI-360 dataset and conduct extensive experiments. The results demonstrate the effectiveness of our approach.