# Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos

arXiv: https://arxiv.org/abs/2407.04384 · HTML: https://mlsanity.com/paper/2407.04384/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild](https://mlsanity.com/paper/2007.12107/) (2020)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [Continuous Surface Embeddings](https://mlsanity.com/paper/2011.12438/) (2020)
- [RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering](https://mlsanity.com/paper/2104.00633/) (2021)
- [Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset](https://mlsanity.com/paper/2206.15436/) (2022)
- [Zero-Shot Category-Level Object Pose Estimation](https://mlsanity.com/paper/2204.03635/) (2022)
- [Robust Object Detection under Occlusion with Context-Aware CompositionalNets](https://mlsanity.com/paper/2005.11643/) (2020)
- [Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusion](https://mlsanity.com/paper/2003.04490/) (2020)
- [StarMap for Category-Agnostic Keypoint and Viewpoint Estimation](https://mlsanity.com/paper/1803.09331/) (2018)
- [NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation](https://mlsanity.com/paper/2101.12378/) (2021)
- [Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects](https://mlsanity.com/paper/1906.05105/) (2019)
- [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild](https://mlsanity.com/paper/1803.11493/) (2018)
- [VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis](https://mlsanity.com/paper/2205.15401/) (2022)
- [Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild](https://mlsanity.com/paper/2210.07199/) (2022)
- [PoseContrast: Class-Agnostic Object Viewpoint Estimation in the Wild with Pose-Aware Contrastive Learning](https://mlsanity.com/paper/2105.05643/) (2021)
- [You Only Look at One: Category-Level Object Representations for Pose Estimation From a Single Example](https://mlsanity.com/paper/2305.12626/) (2023)
- [Few-Shot Viewpoint Estimation](https://mlsanity.com/paper/1905.04957/) (2019)
- [Inferring 3D Object Pose in RGB-D Images](https://mlsanity.com/paper/1502.04652/) (2015)
- [Category-Level Pose Retrieval with Contrastive Features Learnt with Occlusion Augmentation](https://mlsanity.com/paper/2208.06195/) (2022)

## Built on by (most cited first, 5 of 5)

- [Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space](https://mlsanity.com/paper/2504.21749/) (2025)
- [4D-Animal: Freely Reconstructing Animatable 3D Animals from Videos](https://mlsanity.com/paper/2507.10437/) (2025)
- [Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation](https://mlsanity.com/paper/2410.03858/) (2024)
- [Every9D-21M: Large-Scale Real-World 9D Canonicalization of Everyday Objects](https://mlsanity.com/paper/2605.28270/) (2026)
- [Emergence of a Shared Canonical Object Frame from In-the-Wild Videos](https://mlsanity.com/paper/2606.30058/) (2026)

## Abstract (arXiv)

Category-level 3D pose estimation is a fundamentally important problem in computer vision and robotics, e.g. for embodied agents or to train 3D generative models. However, so far methods that estimate the category-level object pose require either large amounts of human annotations, CAD models or input from RGB-D sensors. In contrast, we tackle the problem of learning to estimate the category-level 3D pose only from casually taken object-centric videos without human supervision. We propose a two-step pipeline: First, we introduce a multi-view alignment procedure that determines canonical camera poses across videos with a novel and robust cyclic distance formulation for geometric and appearance matching using reconstructed coarse meshes and DINOv2 features. In a second step, the canonical poses and reconstructed meshes enable us to train a model for 3D pose estimation from a single image. In particular, our model learns to estimate dense correspondences between images and a prototypical 3D template by predicting, for each pixel in a 2D image, a feature vector of the corresponding vertex in the template mesh. We demonstrate that our method outperforms all baselines at the unsupervised alignment of object-centric videos by a large margin and provides faithful and robust predictions in-the-wild. Our code and data is available at https://github.com/GenIntel/uns-obj-pose3d.