# Unsupervised Learning of Shape and Pose with Differentiable Point Clouds

arXiv: https://arxiv.org/abs/1810.09381 · HTML: https://mlsanity.com/paper/1810.09381/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2018.

## Built on by (most cited first, 30 of 89)

- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Pixel-GS: Density Control with Pixel-aware Gradient for 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15530/) (2024)
- [ShaRF: Shape-conditioned Radiance Fields from a Single View](https://mlsanity.com/paper/2102.08860/) (2021)
- [Unsupervised Learning of 3D Object Categories from Videos in the Wild](https://mlsanity.com/paper/2103.16552/) (2021)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [Surface Reconstruction from Point Clouds by Learning Predictive Context Priors](https://mlsanity.com/paper/2204.11015/) (2022)
- [Style-based Point Generator with Adversarial Rendering for Point Cloud Completion](https://mlsanity.com/paper/2103.02535/) (2021)
- [Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation](https://mlsanity.com/paper/2111.00190/) (2021)
- [UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Rendering](https://mlsanity.com/paper/2102.11870/) (2021)
- [Learning Generative Models of Textured 3D Meshes from Real-World Images](https://mlsanity.com/paper/2103.15627/) (2021)
- [Approximate Differentiable Rendering with Algebraic Surfaces](https://mlsanity.com/paper/2207.10606/) (2022)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Cross-modal Learning for Image-Guided Point Cloud Shape Completion](https://mlsanity.com/paper/2209.09552/) (2022)
- [Reconstructing Hand-Held Objects from Monocular Video](https://mlsanity.com/paper/2211.16835/) (2022)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2203.10339/) (2022)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs](https://mlsanity.com/paper/2306.05410/) (2023)
- [Flexible Techniques for Differentiable Rendering with 3D Gaussians](https://mlsanity.com/paper/2308.14737/) (2023)
- [Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching](https://mlsanity.com/paper/2108.03746/) (2021)
- [Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstruction](https://mlsanity.com/paper/2204.03642/) (2022)
- [Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency](https://mlsanity.com/paper/2204.10310/) (2022)
- [CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation](https://mlsanity.com/paper/2311.01447/) (2023)
- [Representing 3D Shapes with Probabilistic Directed Distance Fields](https://mlsanity.com/paper/2112.05300/) (2021)
- [Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images](https://mlsanity.com/paper/2204.09866/) (2022)

## Abstract (arXiv)

We address the problem of learning accurate 3D shape and camera pose from a collection of unlabeled category-specific images. We train a convolutional network to predict both the shape and the pose from a single image by minimizing the reprojection error: given several views of an object, the projections of the predicted shapes to the predicted camera poses should match the provided views. To deal with pose ambiguity, we introduce an ensemble of pose predictors which we then distill to a single "student" model. To allow for efficient learning of high-fidelity shapes, we represent the shapes by point clouds and devise a formulation allowing for differentiable projection of these. Our experiments show that the distilled ensemble of pose predictors learns to estimate the pose accurately, while the point cloud representation allows to predict detailed shape models. The supplementary video can be found at https://www.youtube.com/watch?v=LuIGovKeo60