# RGBTrack: Fast, Robust Depth-Free 6D Pose Estimation and Tracking

arXiv: https://arxiv.org/abs/2506.17119 · HTML: https://mlsanity.com/paper/2506.17119/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [OnePose: One-Shot Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2205.12257/) (2022)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation](https://mlsanity.com/paper/2311.15707/) (2023)
- [OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models](https://mlsanity.com/paper/2301.07673/) (2023)
- [GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence](https://mlsanity.com/paper/2311.14155/) (2023)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [Single-Stage 6D Object Pose Estimation](https://mlsanity.com/paper/1911.08324/) (2019)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions](https://mlsanity.com/paper/2203.17234/) (2022)
- [ClearPose: Large-scale Transparent Object Dataset and Benchmark](https://mlsanity.com/paper/2203.03890/) (2022)
- [ZS6D: Zero-shot 6D Object Pose Estimation using Vision Transformers](https://mlsanity.com/paper/2309.11986/) (2023)

## Built on by (most cited first, 6 of 6)

- [ByteLoom: Weaving Geometry-Consistent Human-Object Interactions through Progressive Curriculum Learning](https://mlsanity.com/paper/2512.22854/) (2025)
- [Mind the Shape Gap: A Benchmark and Baseline for Deformation-Aware 6D Pose Estimation of Agricultural Produce](https://mlsanity.com/paper/2603.27429/) (2026)
- [Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions](https://mlsanity.com/paper/2607.23468/) (2026)
- [RRTrack: Robust and Recoverable Object 6D Pose Tracking for Dynamic Scenes](https://mlsanity.com/paper/2607.23669/) (2026)
- [HAT: Hypothesis-Anchored Tracking for Video Monocular Spacecraft Pose Estimation](https://mlsanity.com/paper/2609.21597/) (2026)
- [GRC-Pose: Generation-Reconstruction Correspondence for Prior-Free 6D Object Pose Tracking](https://mlsanity.com/paper/2609.39116/) (2026)

## Abstract (arXiv)

We introduce a robust framework, RGBTrack, for real-time 6D pose estimation and tracking that operates solely on RGB data, thereby eliminating the need for depth input for such dynamic and precise object pose tracking tasks. Building on the FoundationPose architecture, we devise a novel binary search strategy combined with a render-and-compare mechanism to efficiently infer depth and generate robust pose hypotheses from true-scale CAD models. To maintain stable tracking in dynamic scenarios, including rapid movements and occlusions, RGBTrack integrates state-of-the-art 2D object tracking (XMem) with a Kalman filter and a state machine for proactive object pose recovery. In addition, RGBTrack's scale recovery module dynamically adapts CAD models of unknown scale using an initial depth estimate, enabling seamless integration with modern generative reconstruction techniques. Extensive evaluations on benchmark datasets demonstrate that RGBTrack's novel depth-free approach achieves competitive accuracy and real-time performance, making it a promising practical solution candidate for application areas including robotics, augmented reality, and computer vision. The source code for our implementation will be made publicly available at https://github.com/GreatenAnoymous/RGBTrack.git.