# DeepI2P: Image-to-Point Cloud Registration via Deep Classification

arXiv: https://arxiv.org/abs/2104.03501 · HTML: https://mlsanity.com/paper/2104.03501/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [SO-Net: Self-Organizing Network for Point Cloud Analysis](https://mlsanity.com/paper/1803.04249/) (2018)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [PPFNet: Global Context Aware Local Features for Robust 3D Point Matching](https://mlsanity.com/paper/1802.02669/) (2018)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors](https://mlsanity.com/paper/1808.10322/) (2018)
- [3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration](https://mlsanity.com/paper/1807.09413/) (2018)
- [USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds](https://mlsanity.com/paper/1904.00229/) (2019)
- [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019)
- [LCD: Learned Cross-Domain Descriptors for 2D-3D Matching](https://mlsanity.com/paper/1911.09326/) (2019)
- [Global visual localization in LiDAR-maps through shared 2D-3D embedding space](https://mlsanity.com/paper/1910.04871/) (2019)
- [Introduction to Camera Pose Estimation with Deep Learning](https://mlsanity.com/paper/1907.05272/) (2019)
- [Monocular Camera Localization in Prior LiDAR Maps with 2D-3D Line Correspondences](https://mlsanity.com/paper/2004.00740/) (2020)

## Built on by (most cited first, 30 of 59)

- [CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence](https://mlsanity.com/paper/2207.05483/) (2022)
- [Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching](https://mlsanity.com/paper/2312.04060/) (2023)
- [FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators](https://mlsanity.com/paper/2310.03420/) (2023)
- [A Conditional Denoising Diffusion Probabilistic Model for Point Cloud Upsampling](https://mlsanity.com/paper/2312.02719/) (2023)
- [CSDN: Cross-modal Shape-transfer Dual-refinement Network for Point Cloud Completion](https://mlsanity.com/paper/2208.00751/) (2022)
- [ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairs](https://mlsanity.com/paper/2207.00826/) (2022)
- [(LC)2: LiDAR-Camera Loop Constraints For Cross-Modal Place Recognition](https://mlsanity.com/paper/2304.08660/) (2023)
- [Bridge 2D-3D: Uncertainty-aware Hierarchical Registration Network with Domain Alignment](https://mlsanity.com/paper/2504.01641/) (2025)
- [LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment](https://mlsanity.com/paper/2410.12269/) (2024)
- [CoFiI2P: Coarse-to-Fine Correspondences for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2309.14660/) (2023)
- [E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning](https://mlsanity.com/paper/2311.18433/) (2023)
- [End-to-end 2D-3D Registration between Image and LiDAR Point Cloud for Vehicle Localization](https://mlsanity.com/paper/2306.11346/) (2023)
- [EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization](https://mlsanity.com/paper/2309.07471/) (2023)
- [Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Prior](https://mlsanity.com/paper/2507.06651/) (2025)
- [Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration](https://mlsanity.com/paper/2401.12452/) (2024)
- [Extrinsic Camera Calibration with Semantic Segmentation](https://mlsanity.com/paper/2208.03949/) (2022)
- [Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2312.08234/) (2023)
- [VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition](https://mlsanity.com/paper/2403.14594/) (2024)
- [Quantity-Aware Coarse-to-Fine Correspondence for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2307.07142/) (2023)
- [Diff-Reg v1: Diffusion Matching Model for Registration Problem](https://mlsanity.com/paper/2403.19919/) (2024)
- [CMR-Agent: Learning a Cross-Modal Agent for Iterative Image-to-Point Cloud Registration](https://mlsanity.com/paper/2408.02394/) (2024)
- [EdgeRegNet: Edge Feature-based Multimodal Registration Network between Images and LiDAR Point Clouds](https://mlsanity.com/paper/2503.15284/) (2025)
- [Automated Static Camera Calibration with Intelligent Vehicles](https://mlsanity.com/paper/2304.10814/) (2023)
- [Poses as Queries: Image-to-LiDAR Map Localization with Transformers](https://mlsanity.com/paper/2305.04298/) (2023)
- [SOLVR: Submap Oriented LiDAR-Visual Re-Localisation](https://mlsanity.com/paper/2409.10247/) (2024)
- [TrafficLoc: Localizing Traffic Surveillance Cameras in 3D Scenes](https://mlsanity.com/paper/2412.10308/) (2024)
- [EEPNet: Efficient Edge Pixel-based Matching Network for Cross-Modal Dynamic Registration between LiDAR and Camera](https://mlsanity.com/paper/2409.19305/) (2024)
- [Iterative Camera-LiDAR Extrinsic Optimization via Surrogate Diffusion](https://mlsanity.com/paper/2411.10936/) (2024)
- [Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learning](https://mlsanity.com/paper/2508.01184/) (2025)
- [Self-Supervised Cross-Modal Learning for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2509.15882/) (2025)

## Abstract (arXiv)

This paper presents DeepI2P: a novel approach for cross-modality registration between an image and a point cloud. Given an image (e.g. from a rgb-camera) and a general point cloud (e.g. from a 3D Lidar scanner) captured at different locations in the same scene, our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. Learning common feature descriptors to establish correspondences for the registration is inherently challenging due to the lack of appearance and geometric correlations across the two modalities. We circumvent the difficulty by converting the registration problem into a classification and inverse camera projection optimization problem. A classification neural network is designed to label whether the projection of each point in the point cloud is within or beyond the camera frustum. These labeled points are subsequently passed into a novel inverse camera projection solver to estimate the relative pose. Extensive experimental results on Oxford Robotcar and KITTI datasets demonstrate the feasibility of our approach. Our source code is available at https://github.com/lijx10/DeepI2P