# UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Rendering

arXiv: https://arxiv.org/abs/2102.11870 · HTML: https://mlsanity.com/paper/2102.11870/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 81 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 30 of 55)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [PointNetLK: Robust & Efficient Point Cloud Registration using PointNet](https://mlsanity.com/paper/1903.05711/) (2019)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Scaling and Benchmarking Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1905.01235/) (2019)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features](https://mlsanity.com/paper/2003.03164/) (2020)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)

## Built on by (most cited first, 30 of 44)

- [Probing the 3D Awareness of Visual Foundation Models](https://mlsanity.com/paper/2404.08636/) (2024)
- [Lepard: Learning partial point cloud matching in rigid and deformable scenes](https://mlsanity.com/paper/2111.12591/) (2021)
- [How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey](https://mlsanity.com/paper/2402.13255/) (2024)
- [SiLK -- Simple Learned Keypoints](https://mlsanity.com/paper/2304.06194/) (2023)
- [Few-View Object Reconstruction with Unknown Categories and Camera Poses](https://mlsanity.com/paper/2212.04492/) (2022)
- [Planar Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2103.14644/) (2021)
- [The 8-Point Algorithm as an Inductive Bias for Relative Pose Prediction by ViTs](https://mlsanity.com/paper/2208.08988/) (2022)
- [Learning Visual Representations via Language-Guided Sampling](https://mlsanity.com/paper/2302.12248/) (2023)
- [Bootstrap Your Own Correspondences](https://mlsanity.com/paper/2106.00677/) (2021)
- [Unsupervised Deep Probabilistic Approach for Partial Point Cloud Registration](https://mlsanity.com/paper/2303.13290/) (2023)
- [R-PointHop: A Green, Accurate, and Unsupervised Point Cloud Registration Method](https://mlsanity.com/paper/2103.08129/) (2021)
- [PlaneFormers: From Sparse View Planes to 3D Reconstruction](https://mlsanity.com/paper/2208.04307/) (2022)
- [Last-Mile Embodied Visual Navigation](https://mlsanity.com/paper/2211.11746/) (2022)
- [iDF-SLAM: End-to-End RGB-D SLAM with Neural Implicit Mapping and Deep Feature Tracking](https://mlsanity.com/paper/2209.07919/) (2022)
- [Reliable Inlier Evaluation for Unsupervised Point Cloud Registration](https://mlsanity.com/paper/2202.11292/) (2022)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Extend Your Own Correspondences: Unsupervised Distant Point Cloud Registration by Progressive Distance Extension](https://mlsanity.com/paper/2403.03532/) (2024)
- [PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registration](https://mlsanity.com/paper/2308.04782/) (2023)
- [PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometry](https://mlsanity.com/paper/2302.14418/) (2023)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [Improving RGB-D Point Cloud Registration by Learning Multi-scale Local Linear Transformation](https://mlsanity.com/paper/2208.14893/) (2022)
- [Towards Self-Supervised Category-Level Object Pose and Size Estimation](https://mlsanity.com/paper/2203.02884/) (2022)
- [CMHANet: A Cross-Modal Hybrid Attention Network for Point Cloud Registration](https://mlsanity.com/paper/2603.12721/) (2026)
- [From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs](https://mlsanity.com/paper/2502.20389/) (2025)
- [ZeroReg: Zero-Shot Point Cloud Registration with Foundation Models](https://mlsanity.com/paper/2312.03032/) (2023)
- [COTReg:Coupled Optimal Transport based Point Cloud Registration](https://mlsanity.com/paper/2112.14381/) (2021)
- [Self-Supervised Correspondence Estimation via Multiview Registration](https://mlsanity.com/paper/2212.03236/) (2022)
- [RoCNet: 3D Robust Registration of Point-Clouds using Deep Learning](https://mlsanity.com/paper/2303.07963/) (2023)
- [Overlap Bias Matching is Necessary for Point Cloud Registration](https://mlsanity.com/paper/2308.09364/) (2023)

## Abstract (arXiv)

Aligning partial views of a scene into a single whole is essential to understanding one's environment and is a key component of numerous robotics tasks such as SLAM and SfM. Recent approaches have proposed end-to-end systems that can outperform traditional methods by leveraging pose supervision. However, with the rising prevalence of cameras with depth sensors, we can expect a new stream of raw RGB-D data without the annotations needed for supervision. We propose UnsupervisedR&R: an end-to-end unsupervised approach to learning point cloud registration from raw RGB-D video. The key idea is to leverage differentiable alignment and rendering to enforce photometric and geometric consistency between frames. We evaluate our approach on indoor scene datasets and find that we outperform existing traditional approaches with classic and learned descriptors while being competitive with supervised geometric point cloud registration approaches.