# Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions

arXiv: https://arxiv.org/abs/2607.23468 · HTML: https://mlsanity.com/paper/2607.23468/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [CNOS: A Strong Baseline for CAD-based Novel Object Segmentation](https://mlsanity.com/paper/2307.11067/) (2023)
- [6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints](https://mlsanity.com/paper/1910.10750/) (2019)
- [Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects](https://mlsanity.com/paper/2203.05334/) (2022)
- [SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World](https://mlsanity.com/paper/2110.12715/) (2021)
- [A Learned Stereo Depth System for Robotic Manipulation in Homes](https://mlsanity.com/paper/2109.11644/) (2021)
- [ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking](https://mlsanity.com/paper/2111.03821/) (2021)
- [Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames](https://mlsanity.com/paper/2410.12124/) (2024)
- [GoTrack: Generic 6DoF Object Pose Refinement and Tracking](https://mlsanity.com/paper/2506.07155/) (2025)
- [Real-Time 3D Model Tracking in Color and Depth on a Single CPU Core](https://mlsanity.com/paper/1911.10249/) (2019)
- [Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking](https://mlsanity.com/paper/2302.11458/) (2023)
- [RGBTrack: Fast, Robust Depth-Free 6D Pose Estimation and Tracking](https://mlsanity.com/paper/2506.17119/) (2025)
- [VIPose: Real-time Visual-Inertial 6D Object Pose Tracking](https://mlsanity.com/paper/2107.12617/) (2021)
- [DynamicPose: Real-time and Robust 6D Object Pose Tracking for Fast-Moving Cameras and Objects](https://mlsanity.com/paper/2508.11950/) (2025)
- [EARL: Eye-on-Hand Reinforcement Learner for Dynamic Grasping with Active Pose Estimation](https://mlsanity.com/paper/2310.06751/) (2023)

## Abstract (arXiv)

Real-time 6-DoF object pose tracking is essential for many robotics applications, and several approaches exist. Yet even today's approaches remain unreliable under temporary full occlusions and rapid object motions. Once tracking is lost, most methods struggle to detect the failure and recover automatically, often requiring manual re-initialization. In this paper, we address the problem of robust model-based 6-DoF tracking of unseen objects from RGB-D data, especially in scenarios with occlusion and fast motion. We propose a novel method that combines efficient learning-based keypoint matching with optimization-based alignment and introduces a novel failure detection and recovery module. Our system monitors pose reliability, detects tracking divergence or occlusions, and performs a global re-detection and pose estimation step that robustly verifies recovery candidates before resuming tracking. Our evaluation on standard tracking benchmarks and on a new dataset of occluded and fast-moving scenes shows that our method matches state-of-the-art accuracy on easy tracking sequences, maintains high tracking speed at 57.6 frames per second, and provides the most robust tracking performance under challenging conditions. Thus, we believe that our approach is a relevant step forward in robust 6-DoF object tracking from RGB-D data.