# FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators

arXiv: https://arxiv.org/abs/2310.03420 · HTML: https://mlsanity.com/paper/2310.03420/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 72 of its 97 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 72)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)

## Built on by (most cited first, 30 of 31)

- [VistaDream: Sampling multiview consistent images for single-view scene reconstruction](https://mlsanity.com/paper/2410.16892/) (2024)
- [Guided Diffusion from Self-Supervised Diffusion Features](https://mlsanity.com/paper/2312.08825/) (2023)
- [CA-I2P: Channel-Adaptive Registration Network with Global Optimal Selection](https://mlsanity.com/paper/2506.21364/) (2025)
- [Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Prior](https://mlsanity.com/paper/2507.06651/) (2025)
- [ZeroReg: Zero-Shot Point Cloud Registration with Foundation Models](https://mlsanity.com/paper/2312.03032/) (2023)
- [RARE: Refine Any Registration of Pairwise Point Clouds via Zero-Shot Learning](https://mlsanity.com/paper/2507.19950/) (2025)
- [Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis](https://mlsanity.com/paper/2501.02913/) (2025)
- [Diff-Reg v1: Diffusion Matching Model for Registration Problem](https://mlsanity.com/paper/2403.19919/) (2024)
- [Self-Supervised Spatial Correspondence Across Modalities](https://mlsanity.com/paper/2506.03148/) (2025)
- [TrafficLoc: Localizing Traffic Surveillance Cameras in 3D Scenes](https://mlsanity.com/paper/2412.10308/) (2024)
- [Cross-modal feature fusion for robust point cloud registration with ambiguous geometry](https://mlsanity.com/paper/2505.13088/) (2025)
- [Modality-Aware Feature Matching in Visual and Vision-Language Applications: A Comprehensive Survey](https://mlsanity.com/paper/2507.22791/) (2025)
- [GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registration](https://mlsanity.com/paper/2603.26262/) (2026)
- [Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs](https://mlsanity.com/paper/2603.27969/) (2026)
- [F2M-Reg: Unsupervised RGB-D Point Cloud Registration with Frame-to-Model Optimization](https://mlsanity.com/paper/2405.00507/) (2024)
- [TFCT-I2P: Three stream fusion network with color aware transformer for image-to-point cloud registration](https://mlsanity.com/paper/2410.00360/) (2024)
- [Look to Locate: Vision-Based Multisensory Navigation with 3-D Digital Maps for GNSS-Challenged Environments](https://mlsanity.com/paper/2506.19827/) (2025)
- [CRISTAL: Real-time Camera Registration in Static LiDAR Scans using Neural Rendering](https://mlsanity.com/paper/2511.16349/) (2025)
- [Geometry-to-Image Synthesis-Driven Generative Point Cloud Registration](https://mlsanity.com/paper/2512.09407/) (2025)
- [PlanaReLoc: Camera Relocalization in 3D Planar Primitives via Region-Based Structure Matching](https://mlsanity.com/paper/2603.20818/) (2026)
- [C-GenReg: Training-Free 3D Point Cloud Registration by Multi-View-Consistent Geometry-to-Image Generation with Probabilistic Modalities Fusion](https://mlsanity.com/paper/2604.16680/) (2026)
- [Angle-I2P: Angle-Consistent-Aware Hierarchical Attention for Cross-Modality Outlier Rejection](https://mlsanity.com/paper/2605.04541/) (2026)
- [FS-I2P:A Hierarchical Focus-Sweep Registration Network with Dynamically Allocated Depth](https://mlsanity.com/paper/2605.07607/) (2026)
- [Image-to-Point Cloud Registration Made Easy with Rectified Flow-based LiDAR Upsampling](https://mlsanity.com/paper/2607.14639/) (2026)
- [Cross-Coordinate Correspondence Pruning for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2607.17200/) (2026)
- [TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching](https://mlsanity.com/paper/2608.09590/) (2026)
- [Ex-Sim(3)-Reg: 2D-3D Correspondence Pruning via Extended Sim(3) Registration](https://mlsanity.com/paper/2608.28096/) (2026)
- [Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion](https://mlsanity.com/paper/2609.06007/) (2026)
- [Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2609.18088/) (2026)
- [ZIL: Zero-shot Image-to-LiDAR Registration](https://mlsanity.com/paper/2609.22716/) (2026)

## Abstract (arXiv)

Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. However, due to the modality difference between images and points, it is difficult to learn robust and discriminative cross-modality features by existing metric learning methods for feature matching. Instead of applying metric learning on cross-modality data, we propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross-modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric features, we significantly improve the accuracy of the coarse correspondences produced by diffusion features. Extensive experiments demonstrate that without any task-specific training, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6 percent improvement in Inlier Ratio, a three-fold higher Inlier Number, and a 48.6 percent improvement in Registration Recall than existing state-of-the-arts.