# HIPPo: Harnessing Image-to-3D Priors for Model-free Zero-shot 6D Pose Estimation arXiv: https://arxiv.org/abs/2502.10606 ยท HTML: https://mlsanity.com/paper/2502.10606/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. - Builds on 35 of its 46 references that are in the map. - First posted on arXiv 14 Feb 2025. ## Placed in - Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/) ## Builds on (most cited first, 30 of 35) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022) - [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024) - [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017) - [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023) - [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023) - [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024) - [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023) - [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023) - [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017) - [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022) - [OnePose: One-Shot Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2205.12257/) (2022) - [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023) - [SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation](https://mlsanity.com/paper/2311.15707/) (2023) - [OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2301.07673/) (2023) - [GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence](https://mlsanity.com/paper/2311.14155/) (2023) - [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023) - [FS6D: Few-Shot 6D Pose Estimation of Novel Objects](https://mlsanity.com/paper/2203.14628/) (2022) - [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018) - [LEAP: Liberate Sparse-view 3D Modeling from Camera Poses](https://mlsanity.com/paper/2310.01410/) (2023) - [ZS6D: Zero-shot 6D Object Pose Estimation using Vision Transformers](https://mlsanity.com/paper/2309.11986/) (2023) - [MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving](https://mlsanity.com/paper/2309.16715/) (2023) - [Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation](https://mlsanity.com/paper/2403.14279/) (2024) - [UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations](https://mlsanity.com/paper/2411.15355/) (2024) - [VQA-Diff: Exploiting VQA and Diffusion for Zero-Shot Image-to-3D Vehicle Asset Generation in Autonomous Driving](https://mlsanity.com/paper/2407.06516/) (2024) - [Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes](https://mlsanity.com/paper/2208.05067/) (2022) - [Navigation of a Self-Driving Vehicle Using One Fiducial Marker](https://mlsanity.com/paper/2104.12954/) (2021) - [Toward General Object-level Mapping from Sparse Views with 3D Diffusion Priors](https://mlsanity.com/paper/2410.05514/) (2024) ## Built on by (most cited first, 4 of 4) - [Any6D: Model-free 6D Pose Estimation of Novel Objects](https://mlsanity.com/paper/2503.18673/) (2025) - [Orientation Matters: Making 3D Generative Models Orientation-Aligned](https://mlsanity.com/paper/2506.08640/) (2025) - [ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation](https://mlsanity.com/paper/2607.06555/) (2026) - [SI-Diff: A Framework for Learning Search and High-Precision Insertion with a Force-Domain Diffusion Policy](https://mlsanity.com/paper/2605.12247/) (2026) ## Abstract (arXiv) This work focuses on model-free zero-shot 6D object pose estimation for robotics applications. While existing methods can estimate the precise 6D pose of objects, they heavily rely on curated CAD models or reference images, the preparation of which is a time-consuming and labor-intensive process. Moreover, in real-world scenarios, 3D models or reference images may not be available in advance and instant robot reaction is desired. In this work, we propose a novel framework named HIPPo, which eliminates the need for curated CAD models and reference images by harnessing image-to-3D priors from Diffusion Models, enabling model-free zero-shot 6D pose estimation. Specifically, we construct HIPPo Dreamer, a rapid image-to-mesh model built on a multiview Diffusion Model and a 3D reconstruction foundation model. Our HIPPo Dreamer can generate a 3D mesh of any unseen objects from a single glance in just a few seconds. Then, as more observations are acquired, we propose to continuously refine the diffusion prior mesh model by joint optimization of object geometry and appearance. This is achieved by a measurement-guided scheme that gradually replaces the plausible diffusion priors with more reliable online observations. Consequently, HIPPo can instantly estimate and track the 6D pose of a novel object and maintain a complete mesh for immediate robotic applications. Thorough experiments on various benchmarks show that HIPPo outperforms state-of-the-art methods in 6D object pose estimation when prior reference images are limited.