# Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Prior arXiv: https://arxiv.org/abs/2507.06651 ยท HTML: https://mlsanity.com/paper/2507.06651/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. - Builds on 47 of its 70 references that are in the map. - First posted on arXiv 9 Jul 2025. ## Placed in - Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/) ## Builds on (most cited first, 30 of 47) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017) - [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019) - [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017) - [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022) - [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019) - [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021) - [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023) - [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023) - [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018) - [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016) - [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017) - [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019) - [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016) - [PREDATOR: Registration of 3D Point Clouds with Low Overlap](https://mlsanity.com/paper/2011.13005/) (2020) - [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020) - [Geometric Transformer for Fast and Robust Point Cloud Registration](https://mlsanity.com/paper/2202.06688/) (2022) - [D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features](https://mlsanity.com/paper/2003.03164/) (2020) - [Neighbourhood Consensus Networks](https://mlsanity.com/paper/1810.10510/) (2018) - [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018) - [CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration](https://mlsanity.com/paper/2110.14076/) (2021) - [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018) ## Built on by (most cited first, 8 of 8) - [Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs](https://mlsanity.com/paper/2603.27969/) (2026) - [CRISTAL: Real-time Camera Registration in Static LiDAR Scans using Neural Rendering](https://mlsanity.com/paper/2511.16349/) (2025) - [PlanaReLoc: Camera Relocalization in 3D Planar Primitives via Region-Based Structure Matching](https://mlsanity.com/paper/2603.20818/) (2026) - [FS-I2P:A Hierarchical Focus-Sweep Registration Network with Dynamically Allocated Depth](https://mlsanity.com/paper/2605.07607/) (2026) - [Cross-Coordinate Correspondence Pruning for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2607.17200/) (2026) - [GRIP: Gaussian Rendering as a Cross-Modal Bridge for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2609.25966/) (2026) - [A Differentiable Optimization Framework for Registering Sequential Bounding Boxes with Point Cloud Stream](https://mlsanity.com/paper/2609.34103/) (2026) - [OCA: ODE-Driven Cross-Attention for Image-to-Point-Cloud Registration](https://mlsanity.com/paper/2609.36644/) (2026) ## Abstract (arXiv) Learning cross-modal correspondences is essential for image-to-point cloud (I2P) registration. Existing methods achieve this mostly by utilizing metric learning to enforce feature alignment across modalities, disregarding the inherent modality gap between image and point data. Consequently, this paradigm struggles to ensure accurate cross-modal correspondences. To this end, inspired by the cross-modal generation success of recent large diffusion models, we propose Diff2I2P, a fully Differentiable I2P registration framework, leveraging a novel and effective Diffusion prior for bridging the modality gap. Specifically, we propose a Control-Side Score Distillation (CSD) technique to distill knowledge from a depth-conditioned diffusion model to directly optimize the predicted transformation. However, the gradients on the transformation fail to backpropagate onto the cross-modal features due to the non-differentiability of correspondence retrieval and PnP solver. To this end, we further propose a Deformable Correspondence Tuning (DCT) module to estimate the correspondences in a differentiable way, followed by the transformation estimation using a differentiable PnP solver. With these two designs, the Diffusion model serves as a strong prior to guide the cross-modal feature learning of image and point cloud for forming robust correspondences, which significantly improves the registration. Extensive experimental results demonstrate that Diff2I2P consistently outperforms SoTA I2P registration methods, achieving over 7% improvement in registration recall on the 7-Scenes benchmark.