# ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition

arXiv: https://arxiv.org/abs/2403.18762 · HTML: https://mlsanity.com/paper/2403.18762/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018)
- [TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization](https://mlsanity.com/paper/2204.00097/) (2022)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Revealing the Dark Secrets of Masked Image Modeling](https://mlsanity.com/paper/2205.13543/) (2022)
- [OverlapNet: Loop Closing for LiDAR-based SLAM](https://mlsanity.com/paper/2105.11344/) (2021)
- [OverlapTransformer: An Efficient and Rotation-Invariant Transformer Network for LiDAR-Based Place Recognition](https://mlsanity.com/paper/2203.03397/) (2022)
- [SOE-Net: A Self-Attention and Orientation Encoding Network for Point Cloud based Place Recognition](https://mlsanity.com/paper/2011.12430/) (2020)
- [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019)
- [SegMatch: Segment based loop-closure for 3D point clouds](https://mlsanity.com/paper/1609.07720/) (2016)
- [P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching](https://mlsanity.com/paper/2103.01055/) (2021)
- [BEVPlace: Learning LiDAR-based Place Recognition using Bird's Eye View Images](https://mlsanity.com/paper/2302.14325/) (2023)
- [Global visual localization in LiDAR-maps through shared 2D-3D embedding space](https://mlsanity.com/paper/1910.04871/) (2019)
- [BVMatch: Lidar-based Place Recognition Using Bird's-eye View Images](https://mlsanity.com/paper/2109.00317/) (2021)
- [(LC)2: LiDAR-Camera Loop Constraints For Cross-Modal Place Recognition](https://mlsanity.com/paper/2304.08660/) (2023)
- [I2P-Rec: Recognizing Images on Large-scale Point Cloud Maps through Bird's Eye View Projections](https://mlsanity.com/paper/2303.01043/) (2023)
- [Cross-view Geo-localization with Evolving Transformer](https://mlsanity.com/paper/2107.00842/) (2021)
- [Spherical Multi-Modal Place Recognition for Heterogeneous Sensor Systems](https://mlsanity.com/paper/2104.10067/) (2021)

## Built on by (most cited first, 5 of 5)

- [Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching](https://mlsanity.com/paper/2410.06285/) (2024)
- [Range and Bird's Eye View Fused Cross-Modal Visual Place Recognition](https://mlsanity.com/paper/2502.11742/) (2025)
- [RLPR: Radar-to-LiDAR Place Recognition via Two-Stage Asymmetric Cross-Modal Alignment for Autonomous Driving](https://mlsanity.com/paper/2603.07920/) (2026)
- [MAG-VLAQ: Multi-modal Aerial-Ground Query Aggregation for Cross-View Place Recognition](https://mlsanity.com/paper/2605.09418/) (2026)
- [G2IA: Geometry-Guided Instance-Aware Retrieval and Refinement for Cross-Modal Place Recognition](https://mlsanity.com/paper/2606.15287/) (2026)

## Abstract (arXiv)

Place recognition is an important task for robots and autonomous cars to localize themselves and close loops in pre-built maps. While single-modal sensor-based methods have shown satisfactory performance, cross-modal place recognition that retrieving images from a point-cloud database remains a challenging problem. Current cross-modal methods transform images into 3D points using depth estimation for modality conversion, which are usually computationally intensive and need expensive labeled data for depth supervision. In this work, we introduce a fast and lightweight framework to encode images and point clouds into place-distinctive descriptors. We propose an effective Field of View (FoV) transformation module to convert point clouds into an analogous modality as images. This module eliminates the necessity for depth estimation and helps subsequent modules achieve real-time performance. We further design a non-negative factorization-based encoder to extract mutually consistent semantic features between point clouds and images. This encoder yields more distinctive global descriptors for retrieval. Experimental results on the KITTI dataset show that our proposed methods achieve state-of-the-art performance while running in real time. Additional evaluation on the HAOMO dataset covering a 17 km trajectory further shows the practical generalization capabilities. We have released the implementation of our methods as open source at: https://github.com/haomo-ai/ModaLink.git.