mlsanityData to 14 Oct 2026

End-to-end 2D-3D Registration between Image and LiDAR Point Cloud for Vehicle Localization

arXiv 2306.11346 ↗First posted 20 Jun 2023cs.RO cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Robot localization using a built map is essential for a variety of tasks including accurate navigation and mobile manipulation. A popular approach to robot localization is based on image-to-point cloud registration, which combines illumination-invariant LiDAR-based mapping with economical image-based localization. However, the recent works for image-to-point cloud registration either divide the registration into separate modules or project the point cloud to the depth image to register the RGB and depth images. In this paper, we present I2PNet, a novel end-to-end 2D-3D registration network, which directly registers the raw 3D point cloud with the 2D RGB image using differential modules with a united target. The 2D-3D cost volume module for differential 2D-3D association is proposed to bridge feature extraction and pose regression. The soft point-to-pixel correspondence is implicitly constructed on the intrinsic-independent normalized plane in the 2D-3D cost volume module. Moreover, we introduce an outlier mask prediction module to filter the outliers in the 2D-3D association before pose regression. Furthermore, we propose the coarse-to-fine 2D-3D registration architecture to increase localization accuracy. Extensive localization experiments are conducted on the KITTI, nuScenes, M2DGR, Argoverse, Waymo, and Lyft5 datasets. The results demonstrate that I2PNet outperforms the state-of-the-art by a large margin and has a higher efficiency than the previous works. Moreover, we extend the application of I2PNet to the camera-LiDAR online calibration and demonstrate that I2PNet outperforms recent approaches on the online calibration task. Source codes are released at https://github.com/IRMVLab/I2PNet.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

35 of 71 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  7. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  8. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  9. 2013Network In Networkcited by 605
  10. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 585
  11. 2016LIFT: Learned Invariant Feature Transformcited by 306
  12. 2020PREDATOR: Registration of 3D Point Clouds with Low Overlapcited by 290
23 more
  1. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 240
  2. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 232
  3. 2020RPM-Net: Robust Point Matching using Learned Featurescited by 182
  4. 2020D3Feat: Joint Learning of Dense Detection and Description of 3D Local Featurescited by 177
  5. 20183DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registrationcited by 103
  6. 2019DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registrationcited by 77
  7. 2019USIP: Unsupervised Stable Interest Point Detection from 3D Point Cloudscited by 70
  8. 2017RegNet: Multimodal Sensor Registration Using Deep Neural Networkscited by 69
  9. 2021LCDNet: Deep Loop Closure Detection and Point Cloud Registration for LiDAR SLAMcited by 67
  10. 20192D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloudcited by 62
  11. 2019LO-Net: Deep Real-time Lidar Odometrycited by 60
  12. 2021DeepI2P: Image-to-Point Cloud Registration via Deep Classificationcited by 59
  13. 2022CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondencecited by 44
  14. 2021P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matchingcited by 42
  15. 2020Hierarchical Attention Learning of Scene Flow in 3D Point Cloudscited by 35
  16. 2020PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimizationcited by 35
  17. 2022What Matters for 3D Scene Flow Networkcited by 31
  18. 2019Global visual localization in LiDAR-maps through shared 2D-3D embedding spacecited by 28
  19. 2019CMRNet: Camera to LiDAR-Map Registrationcited by 25
  20. 2021Efficient 3D Deep LiDAR Odometrycited by 25
  21. 2021M2DGR: A Multi-sensor and Multi-scenario SLAM Dataset for Ground Robotscited by 25
  22. 2023I2P-Rec: Recognizing Images on Large-scale Point Cloud Maps through Bird's Eye View Projectionscited by 16
  23. 2020CMRNet++: Map and Camera Agnostic Monocular Visual Localization in LiDAR Mapscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 2; 2026: 2.

9 papers.

  1. 2023FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimatorscited by 31
  2. 2024CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibrationcited by 15
  3. 2023Spherical Frustum Sparse Convolution Network for LiDAR Point Cloud Semantic Segmentationcited by 6
  4. 2024VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognitioncited by 5
  5. 2025Multi-modal Loop Closure Detection with Foundation Models in Severely Unstructured Environmentscited by 2
  6. 2024EEPNet: Efficient Edge Pixel-based Matching Network for Cross-Modal Dynamic Registration between LiDAR and Cameracited by 1
  7. 2026GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registrationcited by 1
  8. 2025Cross3DReg: Towards a Large-scale Real-world Cross-source Point Cloud Registration Benchmarkcited by 0
  9. 2026GRIP: Gaussian Rendering as a Cross-Modal Bridge for Image-to-Point Cloud Registrationcited by 0