# Uni6D: A Unified CNN Framework without Projection Breakdown for 6D Pose Estimation

arXiv: https://arxiv.org/abs/2203.14531 · HTML: https://mlsanity.com/paper/2203.14531/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [EPOS: Estimating 6D Pose of Objects with Symmetries](https://mlsanity.com/paper/2004.00605/) (2020)
- [FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation](https://mlsanity.com/paper/2103.02242/) (2021)
- [Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/2001.09322/) (2020)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [HybridPose: 6D Object Pose Estimation under Hybrid Representations](https://mlsanity.com/paper/2001.01869/) (2020)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation](https://mlsanity.com/paper/1912.00416/) (2019)
- [Single-Stage 6D Object Pose Estimation](https://mlsanity.com/paper/1911.08324/) (2019)
- [Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd](https://mlsanity.com/paper/1512.07506/) (2015)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimation](https://mlsanity.com/paper/1804.03959/) (2018)

## Built on by (most cited first, 22 of 22)

- [HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation](https://mlsanity.com/paper/2303.15743/) (2023)
- [Deep Learning-Based Object Pose Estimation: A Comprehensive Survey](https://mlsanity.com/paper/2405.07801/) (2024)
- [Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation](https://mlsanity.com/paper/2308.05438/) (2023)
- [TransNet: Category-Level Transparent Object Pose Estimation](https://mlsanity.com/paper/2208.10002/) (2022)
- [On the representation and methodology for wide and short range head pose estimation](https://mlsanity.com/paper/2401.05807/) (2024)
- [TransPose: 6D Object Pose Estimation with Geometry-Aware Transformer](https://mlsanity.com/paper/2310.16279/) (2023)
- [GIVEPose: Gradual Intra-class Variation Elimination for RGB-based Category-Level Object Pose Estimation](https://mlsanity.com/paper/2503.15110/) (2025)
- [RaSim: A Range-aware High-fidelity RGB-D Data Simulation Pipeline for Real-world Applications](https://mlsanity.com/paper/2404.03962/) (2024)
- [RDPN6D: Residual-based Dense Point-wise Network for 6Dof Object Pose Estimation Based on RGB-D Images](https://mlsanity.com/paper/2405.08483/) (2024)
- [Uni6Dv2: Noise Elimination for 6D Pose Estimation](https://mlsanity.com/paper/2208.06416/) (2022)
- [Digital Twin Tracking Dataset (DTTD): A New RGB+Depth 3D Dataset for Longer-Range Object Tracking Applications](https://mlsanity.com/paper/2302.05991/) (2023)
- [Geo6D: Geometric Constraints Learning for 6D Pose Estimation](https://mlsanity.com/paper/2210.10959/) (2022)
- [TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation](https://mlsanity.com/paper/2303.13477/) (2023)
- [S2P3: Self-Supervised Polarimetric Pose Prediction](https://mlsanity.com/paper/2312.01105/) (2023)
- [TransNet: Transparent Object Manipulation Through Category-Level Pose Estimation](https://mlsanity.com/paper/2307.12400/) (2023)
- [Robust 6DoF Pose Estimation Against Depth Noise and a Comprehensive Evaluation on a Mobile Dataset](https://mlsanity.com/paper/2309.13570/) (2023)
- [Leveraging Positional Encoding for Robust Multi-Reference-Based Object 6D Pose Estimation](https://mlsanity.com/paper/2401.16284/) (2024)
- [Generative 6D Pose Estimation via Conditional Flow Matching](https://mlsanity.com/paper/2602.19719/) (2026)
- [Polarimetric Information for Multi-Modal 6D Pose Estimation of Photometrically Challenging Objects with Limited Data](https://mlsanity.com/paper/2308.10627/) (2023)
- [VLM6D: VLM based 6Dof Pose Estimation based on RGB-D Images](https://mlsanity.com/paper/2511.00120/) (2025)
- [Learning Fine-Grained Correspondence with Cross-Perspective Perception for Open-Vocabulary 6D Object Pose Estimation](https://mlsanity.com/paper/2601.13565/) (2026)
- [ComPose: When to Trust Hands for Object Pose Tracking](https://mlsanity.com/paper/2605.23523/) (2026)

## Abstract (arXiv)

As RGB-D sensors become more affordable, using RGB-D images to obtain high-accuracy 6D pose estimation results becomes a better option. State-of-the-art approaches typically use different backbones to extract features for RGB and depth images. They use a 2D CNN for RGB images and a per-pixel point cloud network for depth data, as well as a fusion network for feature fusion. We find that the essential reason for using two independent backbones is the "projection breakdown" problem. In the depth image plane, the projected 3D structure of the physical world is preserved by the 1D depth value and its built-in 2D pixel coordinate (UV). Any spatial transformation that modifies UV, such as resize, flip, crop, or pooling operations in the CNN pipeline, breaks the binding between the pixel value and UV coordinate. As a consequence, the 3D structure is no longer preserved by a modified depth image or feature. To address this issue, we propose a simple yet effective method denoted as Uni6D that explicitly takes the extra UV data along with RGB-D images as input. Our method has a Unified CNN framework for 6D pose estimation with a single CNN backbone. In particular, the architecture of our method is based on Mask R-CNN with two extra heads, one named RT head for directly predicting 6D pose and the other named abc head for guiding the network to map the visible points to their coordinates in the 3D model as an auxiliary module. This end-to-end approach balances simplicity and accuracy, achieving comparable accuracy with state of the arts and 7.2x faster inference speed on the YCB-Video dataset.