# Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features

arXiv: https://arxiv.org/abs/2209.05624 · HTML: https://mlsanity.com/paper/2209.05624/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving](https://mlsanity.com/paper/2001.03343/) (2020)
- [DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistency](https://mlsanity.com/paper/2103.06526/) (2021)
- [Category Level Object Pose Estimation via Neural Analysis-by-Synthesis](https://mlsanity.com/paper/2008.08145/) (2020)
- [6-DoF Object Pose from Semantic Keypoints](https://mlsanity.com/paper/1703.04670/) (2017)
- [RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering](https://mlsanity.com/paper/2104.00633/) (2021)
- [Robust Object Detection under Occlusion with Context-Aware CompositionalNets](https://mlsanity.com/paper/2005.11643/) (2020)
- [Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusion](https://mlsanity.com/paper/2003.04490/) (2020)
- [StarMap for Category-Agnostic Keypoint and Viewpoint Estimation](https://mlsanity.com/paper/1803.09331/) (2018)
- [NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation](https://mlsanity.com/paper/2101.12378/) (2021)
- [Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition under Occlusion](https://mlsanity.com/paper/2006.15538/) (2020)
- [Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion](https://mlsanity.com/paper/1905.11826/) (2019)
- [Robust Instance Segmentation through Reasoning about Multi-Object Occlusion](https://mlsanity.com/paper/2012.02107/) (2020)
- [Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose](https://mlsanity.com/paper/2110.14213/) (2021)
- [CoKe: Localized Contrastive Learning for Robust Keypoint Detection](https://mlsanity.com/paper/2009.14115/) (2020)

## Built on by (most cited first, 23 of 23)

- [pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://mlsanity.com/paper/2401.14398/) (2024)
- [SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning](https://mlsanity.com/paper/2504.20024/) (2025)
- [Deep Learning-Based Object Pose Estimation: A Comprehensive Survey](https://mlsanity.com/paper/2405.07801/) (2024)
- [SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models](https://mlsanity.com/paper/2505.00788/) (2025)
- [ImageNet3D: Towards General-Purpose Object-Level 3D Understanding](https://mlsanity.com/paper/2406.09613/) (2024)
- [Challenges for Monocular 6D Object Pose Estimation in Robotics](https://mlsanity.com/paper/2307.12172/) (2023)
- [Generating Images with 3D Annotations Using Diffusion Models](https://mlsanity.com/paper/2306.08103/) (2023)
- [Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering](https://mlsanity.com/paper/2406.00622/) (2024)
- [3D-Aware Visual Question Answering about Parts, Poses and Occlusions](https://mlsanity.com/paper/2310.17914/) (2023)
- [OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2304.10266/) (2023)
- [Compositor: Bottom-up Clustering and Compositing for Robust Part and Object Segmentation](https://mlsanity.com/paper/2306.07404/) (2023)
- [iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning](https://mlsanity.com/paper/2407.09271/) (2024)
- [Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation](https://mlsanity.com/paper/2401.10848/) (2024)
- [Reconstructing Animals and the Wild](https://mlsanity.com/paper/2411.18807/) (2024)
- [Video based Object 6D Pose Estimation using Transformers](https://mlsanity.com/paper/2210.13540/) (2022)
- [Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis](https://mlsanity.com/paper/2306.00118/) (2023)
- [Unified Category-Level Object Detection and Pose Estimation from RGB Images using 3D Prototypes](https://mlsanity.com/paper/2508.02157/) (2025)
- [WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion](https://mlsanity.com/paper/2403.19022/) (2024)
- [Open Challenges for Monocular Single-shot 6D Object Pose Estimation](https://mlsanity.com/paper/2302.11827/) (2023)
- [Learning a Category-level Object Pose Estimator without Pose Annotations](https://mlsanity.com/paper/2404.05626/) (2024)
- [DINeMo: Learning Neural Mesh Models with no 3D Annotations](https://mlsanity.com/paper/2503.20220/) (2025)
- [PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views](https://mlsanity.com/paper/2604.22658/) (2026)
- [Emergence of a Shared Canonical Object Frame from In-the-Wild Videos](https://mlsanity.com/paper/2606.30058/) (2026)

## Abstract (arXiv)

We consider the problem of category-level 6D pose estimation from a single RGB image. Our approach represents an object category as a cuboid mesh and learns a generative model of the neural feature activations at each mesh vertex to perform pose estimation through differentiable rendering. A common problem of rendering-based approaches is that they rely on bounding box proposals, which do not convey information about the 3D rotation of the object and are not reliable when objects are partially occluded. Instead, we introduce a coarse-to-fine optimization strategy that utilizes the rendering process to estimate a sparse set of 6D object proposals, which are subsequently refined with gradient-based optimization. The key to enabling the convergence of our approach is a neural feature representation that is trained to be scale- and rotation-invariant using contrastive learning. Our experiments demonstrate an enhanced category-level 6D pose estimation performance compared to prior work, particularly under strong partial occlusion.