# Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking

arXiv: https://arxiv.org/abs/2406.04316 · HTML: https://mlsanity.com/paper/2406.04316/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 33)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation](https://mlsanity.com/paper/2301.07525/) (2023)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds](https://mlsanity.com/paper/2104.03437/) (2021)
- [Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks](https://mlsanity.com/paper/2108.08755/) (2021)
- [Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset](https://mlsanity.com/paper/2206.15436/) (2022)
- [GFPose: Learning 3D Human Pose Prior with Gradient Fields](https://mlsanity.com/paper/2212.08641/) (2022)
- [HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios](https://mlsanity.com/paper/2212.10428/) (2022)
- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation](https://mlsanity.com/paper/2303.15743/) (2023)
- [GenPose: Generative Category-level Object Pose Estimation via Diffusion Models](https://mlsanity.com/paper/2306.10531/) (2023)
- [PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects](https://mlsanity.com/paper/2205.08811/) (2022)
- [IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation](https://mlsanity.com/paper/2303.13479/) (2023)
- [StereOBJ-1M: Large-scale Stereo Image Dataset for 6D Object Pose Estimation](https://mlsanity.com/paper/2109.10115/) (2021)
- [CATRE: Iterative Point Clouds Alignment for Category-level Object Pose Refinement](https://mlsanity.com/paper/2207.08082/) (2022)
- [GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping](https://mlsanity.com/paper/2309.06038/) (2023)
- [RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation](https://mlsanity.com/paper/2310.03478/) (2023)
- [TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification](https://mlsanity.com/paper/2209.00853/) (2022)
- [Score-PA: Score-based 3D Part Assembly](https://mlsanity.com/paper/2309.04220/) (2023)
- [ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking](https://mlsanity.com/paper/2111.03821/) (2021)

## Built on by (most cited first, 30 of 50)

- [STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?](https://mlsanity.com/paper/2503.23765/) (2025)
- [Any6D: Model-free 6D Pose Estimation of Novel Objects](https://mlsanity.com/paper/2503.18673/) (2025)
- [Orient Anything V2: Unifying Orientation and Rotation Understanding](https://mlsanity.com/paper/2601.05573/) (2026)
- [SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model](https://mlsanity.com/paper/2512.10957/) (2025)
- [See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model](https://mlsanity.com/paper/2509.16087/) (2025)
- [From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs](https://mlsanity.com/paper/2512.19683/) (2025)
- [SafeBimanual: Diffusion-based Trajectory Optimization for Safe Bimanual Manipulation](https://mlsanity.com/paper/2508.18268/) (2025)
- [CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation](https://mlsanity.com/paper/2506.09343/) (2025)
- [EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos](https://mlsanity.com/paper/2601.01050/) (2026)
- [PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies](https://mlsanity.com/paper/2602.19710/) (2026)
- [UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models](https://mlsanity.com/paper/2409.20551/) (2024)
- [CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image](https://mlsanity.com/paper/2504.11230/) (2025)
- [MK-Pose: Category-Level Object Pose Estimation via Multimodal-Based Keypoint Learning](https://mlsanity.com/paper/2507.06662/) (2025)
- [Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models](https://mlsanity.com/paper/2605.29074/) (2026)
- [BoxDreamer: Dreaming Box Corners for Generalizable Object Pose Estimation](https://mlsanity.com/paper/2504.07955/) (2025)
- [ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation](https://mlsanity.com/paper/2607.06555/) (2026)
- [Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors](https://mlsanity.com/paper/2410.09467/) (2024)
- [UnPose: Uncertainty-Guided Diffusion Priors for Zero-Shot Pose Estimation](https://mlsanity.com/paper/2508.15972/) (2025)
- [Beyond 'Templates': Category-Agnostic Object Pose, Size, and Shape Estimation from a Single View](https://mlsanity.com/paper/2510.11687/) (2025)
- [AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence](https://mlsanity.com/paper/2604.10579/) (2026)
- [ReSTI: A Source-Grounded Audit and Repair of STI-Bench](https://mlsanity.com/paper/2609.24727/) (2026)
- [Foundation Feature-Driven Online End-Effector Pose Estimation: A Marker-Free and Learning-Free Approach](https://mlsanity.com/paper/2503.14051/) (2025)
- [SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams](https://mlsanity.com/paper/2505.19487/) (2025)
- [T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models](https://mlsanity.com/paper/2506.19498/) (2025)
- [Joint Learning of Pose Regression and Denoising Diffusion with Score Scaling Sampling for Category-level 6D Pose Estimation](https://mlsanity.com/paper/2510.04125/) (2025)
- [COG: Confidence-aware Optimal Geometric Correspondence for Unsupervised Single-reference Novel Object Pose Estimation](https://mlsanity.com/paper/2603.00493/) (2026)
- [CanoVerse: 3D Object Scalable Canonicalization and Dataset for Generation and Pose](https://mlsanity.com/paper/2603.07144/) (2026)
- [MessyKitchens: Contact-rich object-level 3D scene reconstruction](https://mlsanity.com/paper/2603.16868/) (2026)
- [OCH3R: Object-Centric Holistic 3D Reconstruction](https://mlsanity.com/paper/2605.13018/) (2026)
- [DynOPETs: A Versatile Benchmark for Dynamic Object Pose Estimation and Tracking in Moving Camera Scenarios](https://mlsanity.com/paper/2503.19625/) (2025)

## Abstract (arXiv)

6D Object Pose Estimation is a crucial yet challenging task in computer vision, suffering from a significant lack of large-scale datasets. This scarcity impedes comprehensive evaluation of model performance, limiting research advancements. Furthermore, the restricted number of available instances or categories curtails its applications. To address these issues, this paper introduces Omni6DPose, a substantial dataset characterized by its diversity in object categories, large scale, and variety in object materials. Omni6DPose is divided into three main components: ROPE (Real 6D Object Pose Estimation Dataset), which includes 332K images annotated with over 1.5M annotations across 581 instances in 149 categories; SOPE(Simulated 6D Object Pose Estimation Dataset), consisting of 475K images created in a mixed reality setting with depth simulation, annotated with over 5M annotations across 4162 instances in the same 149 categories; and the manually aligned real scanned objects used in both ROPE and SOPE. Omni6DPose is inherently challenging due to the substantial variations and ambiguities. To address this challenge, we introduce GenPose++, an enhanced version of the SOTA category-level pose estimation framework, incorporating two pivotal improvements: Semantic-aware feature extraction and Clustering-based aggregation. Moreover, we provide a comprehensive benchmarking analysis to evaluate the performance of previous methods on this large-scale dataset in the realms of 6D object pose estimation and pose tracking.