# MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images

arXiv: https://arxiv.org/abs/2403.01517 · HTML: https://mlsanity.com/paper/2403.01517/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 70 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [PREDATOR: Registration of 3D Point Clouds with Low Overlap](https://mlsanity.com/paper/2011.13005/) (2020)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [Geometric Transformer for Fast and Robust Point Cloud Registration](https://mlsanity.com/paper/2202.06688/) (2022)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [OnePose: One-Shot Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2205.12257/) (2022)
- [CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration](https://mlsanity.com/paper/2110.14076/) (2021)
- [GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator](https://mlsanity.com/paper/2102.12145/) (2021)
- [FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation](https://mlsanity.com/paper/2103.02242/) (2021)
- [Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images](https://mlsanity.com/paper/2204.10776/) (2022)
- [ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation](https://mlsanity.com/paper/2203.09418/) (2022)

## Built on by (most cited first, 28 of 28)

- [Deep Learning-Based Object Pose Estimation: A Comprehensive Survey](https://mlsanity.com/paper/2405.07801/) (2024)
- [GCE-Pose: Global Context Enhancement for Category-level Object Pose Estimation](https://mlsanity.com/paper/2502.04293/) (2025)
- [Accurate and efficient zero-shot 6D pose estimation with frozen foundation models](https://mlsanity.com/paper/2506.09784/) (2025)
- [UNOPose: Unseen Object Pose Estimation with an Unposed RGB-D Reference Image](https://mlsanity.com/paper/2411.16106/) (2024)
- [Scalable Unseen Objects 6-DoF Absolute Pose Estimation with Robotic Integration](https://mlsanity.com/paper/2503.05578/) (2025)
- [UA-Pose: Uncertainty-Aware 6D Object Pose Estimation and Online Object Completion with Partial References](https://mlsanity.com/paper/2506.07996/) (2025)
- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [XYZ-IBD: Benchmarking Robust 6D Object Pose Estimation under Real-World Industrial Complexity](https://mlsanity.com/paper/2506.00599/) (2025)
- [ConceptPose: Training-Free Zero-Shot Object Pose Estimation using Concept Vectors](https://mlsanity.com/paper/2512.09056/) (2025)
- [RayPose: Ray Bundling Diffusion for Template Views in Unseen 6D Object Pose Estimation](https://mlsanity.com/paper/2510.18521/) (2025)
- [Alignist: CAD-Informed Orientation Distribution Estimation by Fusing Shape and Correspondences](https://mlsanity.com/paper/2409.06683/) (2024)
- [CA-W3D: Leveraging Context-Aware Knowledge for Weakly Supervised Monocular 3D Detection](https://mlsanity.com/paper/2503.04154/) (2025)
- [Foundation Feature-Driven Online End-Effector Pose Estimation: A Marker-Free and Learning-Free Approach](https://mlsanity.com/paper/2503.14051/) (2025)
- [Zero-shot Inexact CAD Model Alignment from a Single Image](https://mlsanity.com/paper/2507.03292/) (2025)
- [SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference](https://mlsanity.com/paper/2509.21927/) (2025)
- [SABER-6D: Shape Representation Based Implicit Object Pose Estimation](https://mlsanity.com/paper/2408.05867/) (2024)
- [DynOPETs: A Versatile Benchmark for Dynamic Object Pose Estimation and Tracking in Moving Camera Scenarios](https://mlsanity.com/paper/2503.19625/) (2025)
- [BioDet: Boosting Industrial Object Detection with Image Preprocessing Strategies](https://mlsanity.com/paper/2510.21000/) (2025)
- [Geo6DPose: Fast Zero-Shot 6D Object Pose Estimation via Geometry-Filtered Feature Matching](https://mlsanity.com/paper/2512.10674/) (2025)
- [PoseGAM: Robust Unseen Object Pose Estimation via Geometry-Aware Multi-View Reasoning](https://mlsanity.com/paper/2512.10840/) (2025)
- [MixRI: Mixing Features of Reference Images for Novel Object Pose Estimation](https://mlsanity.com/paper/2601.06883/) (2026)
- [MAPRPose: Mask-Aware Proposal and Amodal Refinement for Multi-Object 6D Pose Estimation](https://mlsanity.com/paper/2604.20650/) (2026)
- [Pose Anything Anywhere:Model-free Object Poses from Arbitrary References](https://mlsanity.com/paper/2606.23634/) (2026)
- [SOS! : A Streamlined Object-Conditional Transformer for Model-free Segmentation](https://mlsanity.com/paper/2608.15295/) (2026)
- [A Geometry-Driven, Framework-Agnostic Optimization for Object Pose Estimation](https://mlsanity.com/paper/2608.26859/) (2026)
- [G6D: Geometric Learning-Free RGB-D 6D Pose Solver for Robotic Manipulation](https://mlsanity.com/paper/2609.23566/) (2026)
- [LEGAU: Learning Semantic Gaussian Priors for Scalable Category-level Pose Estimation](https://mlsanity.com/paper/2609.35046/) (2026)
- [GRC-Pose: Generation-Reconstruction Correspondence for Prior-Free 6D Object Pose Tracking](https://mlsanity.com/paper/2609.39116/) (2026)

## Abstract (arXiv)

Recent learning methods for object pose estimation require resource-intensive training for each individual object instance or category, hampering their scalability in real applications when confronted with previously unseen objects. In this paper, we propose MatchU, a Fuse-Describe-Match strategy for 6D pose estimation from RGB-D images. MatchU is a generic approach that fuses 2D texture and 3D geometric cues for 6D pose prediction of unseen objects. We rely on learning geometric 3D descriptors that are rotation-invariant by design. By encoding pose-agnostic geometry, the learned descriptors naturally generalize to unseen objects and capture symmetries. To tackle ambiguous associations using 3D geometry only, we fuse additional RGB information into our descriptor. This is achieved through a novel attention-based mechanism that fuses cross-modal information, together with a matching loss that leverages the latent space learned from RGB data to guide the descriptor learning process. Extensive experiments reveal the generalizability of both the RGB-D fusion strategy as well as the descriptor efficacy. Benefiting from the novel designs, MatchU surpasses all existing methods by a significant margin in terms of both accuracy and speed, even without the requirement of expensive re-training or rendering.