# Self-supervised Geometric Perception

arXiv: https://arxiv.org/abs/2103.03114 · HTML: https://mlsanity.com/paper/2103.03114/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 99 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 30 of 55)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features](https://mlsanity.com/paper/2003.03164/) (2020)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [MAGSAC++, a fast, reliable and accurate robust estimator](https://mlsanity.com/paper/1912.05909/) (2019)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)

## Built on by (most cited first, 26 of 26)

- [Lepard: Learning partial point cloud matching in rigid and deformable scenes](https://mlsanity.com/paper/2111.12591/) (2021)
- [GIM: Learning Generalizable Image Matcher From Internet Videos](https://mlsanity.com/paper/2402.11095/) (2024)
- [Unsupervised Deep Probabilistic Approach for Partial Point Cloud Registration](https://mlsanity.com/paper/2303.13290/) (2023)
- [Certifiably Optimal Outlier-Robust Geometric Perception: Semidefinite Relaxations and Scalable Global Optimization](https://mlsanity.com/paper/2109.03349/) (2021)
- [Advances in Global Solvers for 3D Vision](https://mlsanity.com/paper/2602.14662/) (2026)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception](https://mlsanity.com/paper/2110.00511/) (2021)
- [Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids](https://mlsanity.com/paper/2305.13220/) (2023)
- [WSDesc: Weakly Supervised 3D Local Descriptor Learning for Point Cloud Registration](https://mlsanity.com/paper/2108.02740/) (2021)
- [Dynamical Pose Estimation](https://mlsanity.com/paper/2103.06182/) (2021)
- [MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints](https://mlsanity.com/paper/2109.06768/) (2021)
- [Bundle Adjustment in the Eager Mode](https://mlsanity.com/paper/2409.12190/) (2024)
- [TUSK: Task-Agnostic Unsupervised Keypoints](https://mlsanity.com/paper/2206.08460/) (2022)
- [Self-Supervised Correspondence Estimation via Multiview Registration](https://mlsanity.com/paper/2212.03236/) (2022)
- [Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning](https://mlsanity.com/paper/2206.02856/) (2022)
- [Unsupervised Point Cloud Registration with Self-Distillation](https://mlsanity.com/paper/2409.07558/) (2024)
- [Digging Into Self-Supervised Learning of Feature Descriptors](https://mlsanity.com/paper/2110.04773/) (2021)
- [Patch-NetVLAD+: Learned patch descriptor and weighted matching strategy for place recognition](https://mlsanity.com/paper/2202.05738/) (2022)
- [Self-Supervised Place Recognition by Refining Temporal and Featural Pseudo Labels from Panoramic Data](https://mlsanity.com/paper/2208.09315/) (2022)
- [Revisiting LiDAR Registration and Reconstruction: A Range Image Perspective](https://mlsanity.com/paper/2112.02779/) (2021)
- [3D Registration in 30 Years: A Survey](https://mlsanity.com/paper/2412.13735/) (2024)
- [Box Pose and Shape Estimation and Domain Adaptation for Large-Scale Warehouse Automation](https://mlsanity.com/paper/2507.00984/) (2025)
- [Mining and Transferring Feature-Geometry Coherence for Unsupervised Point Cloud Registration](https://mlsanity.com/paper/2411.01870/) (2024)
- [DDM-NET: End-to-end learning of keypoint feature Detection, Description and Matching for 3D localization](https://mlsanity.com/paper/2212.04575/) (2022)
- [A Probabilistic Relaxation of the Two-Stage Object Pose Estimation Paradigm](https://mlsanity.com/paper/2306.00892/) (2023)
- [Unsupervised Point Cloud Registration via Training-Time Semantic Guidance](https://mlsanity.com/paper/2609.15228/) (2026)

## Abstract (arXiv)

We present self-supervised geometric perception (SGP), the first general framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels (e.g., camera poses, rigid transformations). Our first contribution is to formulate geometric perception as an optimization problem that jointly optimizes the feature descriptor and the geometric models given a large corpus of visual measurements (e.g., images, point clouds). Under this optimization formulation, we show that two important streams of research in vision, namely robust model fitting and deep feature learning, correspond to optimizing one block of the unknown variables while fixing the other block. This analysis naturally leads to our second contribution -- the SGP algorithm that performs alternating minimization to solve the joint optimization. SGP iteratively executes two meta-algorithms: a teacher that performs robust model fitting given learned features to generate geometric pseudo-labels, and a student that performs deep feature learning under noisy supervision of the pseudo-labels. As a third contribution, we apply SGP to two perception problems on large-scale real datasets, namely relative camera pose estimation on MegaDepth and point cloud registration on 3DMatch. We demonstrate that SGP achieves state-of-the-art performance that is on-par or superior to the supervised oracles trained using ground-truth labels.