# EEPNet-V2: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image

arXiv: https://arxiv.org/abs/2503.15285 · HTML: https://mlsanity.com/paper/2503.15285/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)
- [Depth-aware CNN for RGB-D Segmentation](https://mlsanity.com/paper/1803.06791/) (2018)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019)
- [DeepI2P: Image-to-Point Cloud Registration via Deep Classification](https://mlsanity.com/paper/2104.03501/) (2021)
- [Is Geometry Enough for Matching in Visual Localization?](https://mlsanity.com/paper/2203.12979/) (2022)
- [CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence](https://mlsanity.com/paper/2207.05483/) (2022)
- [P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching](https://mlsanity.com/paper/2103.01055/) (2021)
- [Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching](https://mlsanity.com/paper/2312.04060/) (2023)
- [LCD: Learned Cross-Domain Descriptors for 2D-3D Matching](https://mlsanity.com/paper/1911.09326/) (2019)
- [SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera](https://mlsanity.com/paper/2003.04260/) (2020)
- [EEPNet: Efficient Edge Pixel-based Matching Network for Cross-Modal Dynamic Registration between LiDAR and Camera](https://mlsanity.com/paper/2409.19305/) (2024)

## Abstract (arXiv)

The primary requirement for cross-modal data fusion is the precise alignment of data from different sensors. However, the calibration between LiDAR point clouds and camera images is typically time-consuming and needs external calibration board or specific environmental features. Cross-modal registration effectively solves this problem by aligning the data directly without requiring external calibration. However, due to the domain gap between the point cloud and the image, existing methods rarely achieve satisfactory registration accuracy while maintaining real-time performance. To address this issue, we propose a framework that projects point clouds into several 2D representations for matching with camera images, which not only leverages the geometric characteristic of LiDAR point clouds effectively but also bridge the domain gap between the point cloud and image. Moreover, to tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network to effectively extract features from both camera images and the projection maps of LiDAR point cloud. Additionally, we propose a patch-to-pixel matching network to provide more effective supervision and achieve high accuracy. We validate the performance of our model through experiments on the KITTI and nuScenes datasets. Experimental results demonstrate the the proposed method achieves real-time performance and extremely high registration accuracy. Specifically, on the KITTI dataset, our model achieves a registration accuracy rate of over 99%. Our code is released at: https://github.com/ESRSchao/EEPNet-V2.