# ZS6D: Zero-shot 6D Object Pose Estimation using Vision Transformers

arXiv: https://arxiv.org/abs/2309.11986 · HTML: https://mlsanity.com/paper/2309.11986/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [EPOS: Estimating 6D Pose of Objects with Symmetries](https://mlsanity.com/paper/2004.00605/) (2020)
- [GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator](https://mlsanity.com/paper/2102.12145/) (2021)
- [ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation](https://mlsanity.com/paper/2203.09418/) (2022)
- [CNOS: A Strong Baseline for CAD-based Novel Object Segmentation](https://mlsanity.com/paper/2307.11067/) (2023)
- [OSOP: A Multi-Stage One Shot Object Pose Estimation Framework](https://mlsanity.com/paper/2203.15533/) (2022)
- [Zero-Shot Category-Level Object Pose Estimation](https://mlsanity.com/paper/2204.03635/) (2022)
- [Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions](https://mlsanity.com/paper/2203.17234/) (2022)
- [Multi-path Learning for Object Pose Estimation Across Domains](https://mlsanity.com/paper/1908.00151/) (2019)
- [Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image](https://mlsanity.com/paper/2109.06161/) (2021)
- [POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference](https://mlsanity.com/paper/2305.15727/) (2023)
- [Towards Bridging the Space Domain Gap for Satellite Pose Estimation using Event Sensing](https://mlsanity.com/paper/2209.11945/) (2022)
- [PyraPose: Feature Pyramids for Fast and Accurate Object Pose Estimation under Domain Shift](https://mlsanity.com/paper/2010.16117/) (2020)
- [COPE: End-to-end trainable Constant Runtime Object Pose Estimation](https://mlsanity.com/paper/2208.08807/) (2022)

## Built on by (most cited first, 30 of 45)

- [GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence](https://mlsanity.com/paper/2311.14155/) (2023)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [FreeZe: Training-free zero-shot 6D pose estimation with geometric and vision foundation models](https://mlsanity.com/paper/2312.00947/) (2023)
- [Deep Learning-Based Object Pose Estimation: A Comprehensive Survey](https://mlsanity.com/paper/2405.07801/) (2024)
- [Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations](https://mlsanity.com/paper/2507.00990/) (2025)
- [GCE-Pose: Global Context Enhancement for Category-level Object Pose Estimation](https://mlsanity.com/paper/2502.04293/) (2025)
- [One2Any: One-Reference 6D Pose Estimation for Any Object](https://mlsanity.com/paper/2505.04109/) (2025)
- [Co-op: Correspondence-based Novel Object Pose Estimation](https://mlsanity.com/paper/2503.17731/) (2025)
- [Accurate and efficient zero-shot 6D pose estimation with frozen foundation models](https://mlsanity.com/paper/2506.09784/) (2025)
- [Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation](https://mlsanity.com/paper/2403.14279/) (2024)
- [GoTrack: Generic 6DoF Object Pose Refinement and Tracking](https://mlsanity.com/paper/2506.07155/) (2025)
- [UA-Pose: Uncertainty-Aware 6D Object Pose Estimation and Online Object Completion with Partial References](https://mlsanity.com/paper/2506.07996/) (2025)
- [Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learning](https://mlsanity.com/paper/2411.03294/) (2024)
- [RGBTrack: Fast, Robust Depth-Free 6D Pose Estimation and Tracking](https://mlsanity.com/paper/2506.17119/) (2025)
- [Diffusion Features for Zero-Shot 6DoF Object Pose Estimation](https://mlsanity.com/paper/2411.16668/) (2024)
- [HIPPo: Harnessing Image-to-3D Priors for Model-free Zero-shot 6D Pose Estimation](https://mlsanity.com/paper/2502.10606/) (2025)
- [Distilling 3D distinctive local descriptors for 6D pose estimation](https://mlsanity.com/paper/2503.15106/) (2025)
- [RayPose: Ray Bundling Diffusion for Template Views in Unseen 6D Object Pose Estimation](https://mlsanity.com/paper/2510.18521/) (2025)
- [ZeroBP: Learning Position-Aware Correspondence for Zero-shot 6D Pose Estimation in Bin-Picking](https://mlsanity.com/paper/2502.01004/) (2025)
- [Reasoning and Learning a Perceptual Metric for Self-Training of Reflective Objects in Bin-Picking with a Low-cost Camera](https://mlsanity.com/paper/2503.20207/) (2025)
- [IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools](https://mlsanity.com/paper/2602.05555/) (2026)
- [Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching](https://mlsanity.com/paper/2411.15860/) (2024)
- [Multi-Modal 3D Mesh Reconstruction from Images and Text](https://mlsanity.com/paper/2503.07190/) (2025)
- [Foundation Feature-Driven Online End-Effector Pose Estimation: A Marker-Free and Learning-Free Approach](https://mlsanity.com/paper/2503.14051/) (2025)
- [RefPose: Leveraging Reference Geometric Correspondences for Accurate 6D Pose Estimation of Unseen Objects](https://mlsanity.com/paper/2505.10841/) (2025)
- [Zero-shot Inexact CAD Model Alignment from a Single Image](https://mlsanity.com/paper/2507.03292/) (2025)
- [PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation](https://mlsanity.com/paper/2508.05976/) (2025)
- [OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding](https://mlsanity.com/paper/2511.12614/) (2025)
- [Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception](https://mlsanity.com/paper/2602.04343/) (2026)
- [PoseIDON: 6DoF Pose Estimation with Foundation Model Features for Marine Sediment Burial Mapping](https://mlsanity.com/paper/2506.10386/) (2025)

## Abstract (arXiv)

As robotic systems increasingly encounter complex and unconstrained real-world scenarios, there is a demand to recognize diverse objects. The state-of-the-art 6D object pose estimation methods rely on object-specific training and therefore do not generalize to unseen objects. Recent novel object pose estimation methods are solving this issue using task-specific fine-tuned CNNs for deep template matching. This adaptation for pose estimation still requires expensive data rendering and training procedures. MegaPose for example is trained on a dataset consisting of two million images showing 20,000 different objects to reach such generalization capabilities. To overcome this shortcoming we introduce ZS6D, for zero-shot novel object 6D pose estimation. Visual descriptors, extracted using pre-trained Vision Transformers (ViT), are used for matching rendered templates against query images of objects and for establishing local correspondences. These local correspondences enable deriving geometric correspondences and are used for estimating the object's 6D pose with RANSAC-based PnP. This approach showcases that the image descriptors extracted by pre-trained ViTs are well-suited to achieve a notable improvement over two state-of-the-art novel object 6D pose estimation methods, without the need for task-specific fine-tuning. Experiments are performed on LMO, YCBV, and TLESS. In comparison to one of the two methods we improve the Average Recall on all three datasets and compared to the second method we improve on two datasets.