# End-to-end 2D-3D Registration between Image and LiDAR Point Cloud for Vehicle Localization arXiv: https://arxiv.org/abs/2306.11346 ยท HTML: https://mlsanity.com/paper/2306.11346/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 71 references that are in the map. - First posted on arXiv 20 Jun 2023. ## Builds on (most cited first, 30 of 35) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017) - [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019) - [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013) - [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019) - [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016) - [PREDATOR: Registration of 3D Point Clouds with Low Overlap](https://mlsanity.com/paper/2011.13005/) (2020) - [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020) - [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017) - [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020) - [D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features](https://mlsanity.com/paper/2003.03164/) (2020) - [3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration](https://mlsanity.com/paper/1807.09413/) (2018) - [DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration](https://mlsanity.com/paper/1905.04153/) (2019) - [USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds](https://mlsanity.com/paper/1904.00229/) (2019) - [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017) - [LCDNet: Deep Loop Closure Detection and Point Cloud Registration for LiDAR SLAM](https://mlsanity.com/paper/2103.05056/) (2021) - [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019) - [LO-Net: Deep Real-time Lidar Odometry](https://mlsanity.com/paper/1904.08242/) (2019) - [DeepI2P: Image-to-Point Cloud Registration via Deep Classification](https://mlsanity.com/paper/2104.03501/) (2021) - [CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence](https://mlsanity.com/paper/2207.05483/) (2022) - [P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching](https://mlsanity.com/paper/2103.01055/) (2021) - [Hierarchical Attention Learning of Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/2010.05762/) (2020) - [PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimization](https://mlsanity.com/paper/2012.00972/) (2020) - [What Matters for 3D Scene Flow Network](https://mlsanity.com/paper/2207.09143/) (2022) - [Global visual localization in LiDAR-maps through shared 2D-3D embedding space](https://mlsanity.com/paper/1910.04871/) (2019) ## Built on by (most cited first, 9 of 9) - [FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators](https://mlsanity.com/paper/2310.03420/) (2023) - [CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration](https://mlsanity.com/paper/2402.00129/) (2024) - [Spherical Frustum Sparse Convolution Network for LiDAR Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2311.17491/) (2023) - [VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition](https://mlsanity.com/paper/2403.14594/) (2024) - [Multi-modal Loop Closure Detection with Foundation Models in Severely Unstructured Environments](https://mlsanity.com/paper/2511.05404/) (2025) - [EEPNet: Efficient Edge Pixel-based Matching Network for Cross-Modal Dynamic Registration between LiDAR and Camera](https://mlsanity.com/paper/2409.19305/) (2024) - [GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registration](https://mlsanity.com/paper/2603.26262/) (2026) - [Cross3DReg: Towards a Large-scale Real-world Cross-source Point Cloud Registration Benchmark](https://mlsanity.com/paper/2509.06456/) (2025) - [GRIP: Gaussian Rendering as a Cross-Modal Bridge for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2609.25966/) (2026) ## Abstract (arXiv) Robot localization using a built map is essential for a variety of tasks including accurate navigation and mobile manipulation. A popular approach to robot localization is based on image-to-point cloud registration, which combines illumination-invariant LiDAR-based mapping with economical image-based localization. However, the recent works for image-to-point cloud registration either divide the registration into separate modules or project the point cloud to the depth image to register the RGB and depth images. In this paper, we present I2PNet, a novel end-to-end 2D-3D registration network, which directly registers the raw 3D point cloud with the 2D RGB image using differential modules with a united target. The 2D-3D cost volume module for differential 2D-3D association is proposed to bridge feature extraction and pose regression. The soft point-to-pixel correspondence is implicitly constructed on the intrinsic-independent normalized plane in the 2D-3D cost volume module. Moreover, we introduce an outlier mask prediction module to filter the outliers in the 2D-3D association before pose regression. Furthermore, we propose the coarse-to-fine 2D-3D registration architecture to increase localization accuracy. Extensive localization experiments are conducted on the KITTI, nuScenes, M2DGR, Argoverse, Waymo, and Lyft5 datasets. The results demonstrate that I2PNet outperforms the state-of-the-art by a large margin and has a higher efficiency than the previous works. Moreover, we extend the application of I2PNet to the camera-LiDAR online calibration and demonstrate that I2PNet outperforms recent approaches on the online calibration task. Source codes are released at https://github.com/IRMVLab/I2PNet.