# SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World

arXiv: https://arxiv.org/abs/2110.12715 · HTML: https://mlsanity.com/paper/2110.12715/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [A Region-based Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking](https://mlsanity.com/paper/1807.02087/) (2018)
- [Deep 6-DOF Tracking](https://mlsanity.com/paper/1703.09771/) (2017)
- [Real-Time 3D Model Tracking in Color and Depth on a Single CPU Core](https://mlsanity.com/paper/1911.10249/) (2019)
- [Combining 3D Model Contour Energy and Keypoints for Object Tracking](https://mlsanity.com/paper/2002.01379/) (2020)
- [Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2105.14391/) (2021)

## Built on by (most cited first, 19 of 19)

- [Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects](https://mlsanity.com/paper/2203.05334/) (2022)
- [A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds](https://mlsanity.com/paper/2203.04232/) (2022)
- [6D Object Pose Estimation from Approximate 3D Models for Orbital Robotics](https://mlsanity.com/paper/2303.13241/) (2023)
- [GoTrack: Generic 6DoF Object Pose Refinement and Tracking](https://mlsanity.com/paper/2506.07155/) (2025)
- [Large-displacement 3D Object Tracking with Hybrid Non-local Optimization](https://mlsanity.com/paper/2207.12620/) (2022)
- [Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking](https://mlsanity.com/paper/2302.11458/) (2023)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [DynamicPose: Real-time and Robust 6D Object Pose Tracking for Fast-Moving Cameras and Objects](https://mlsanity.com/paper/2508.11950/) (2025)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [A Multi-body Tracking Framework - From Rigid Objects to Kinematic Structures](https://mlsanity.com/paper/2208.01502/) (2022)
- [For A More Comprehensive Evaluation of 6DoF Object Pose Tracking](https://mlsanity.com/paper/2309.07796/) (2023)
- [Event6D: Event-based Novel Object 6D Pose Tracking](https://mlsanity.com/paper/2603.28045/) (2026)
- [YCB-Ev 1.1: Event-vision dataset for 6DoF object pose estimation](https://mlsanity.com/paper/2309.08482/) (2023)
- [Robust 6DoF Pose Tracking Considering Contour and Interior Correspondence Uncertainty for AR Assembly Guidance](https://mlsanity.com/paper/2502.11971/) (2025)
- [Active Control Points-based 6DoF Pose Tracking for Industrial Metal Objects](https://mlsanity.com/paper/2507.01478/) (2025)
- [Temporally Consistent Object 6D Pose Estimation for Robot Control](https://mlsanity.com/paper/2605.02708/) (2026)
- [Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions](https://mlsanity.com/paper/2607.23468/) (2026)
- [HAT: Hypothesis-Anchored Tracking for Video Monocular Spacecraft Pose Estimation](https://mlsanity.com/paper/2609.21597/) (2026)

## Abstract (arXiv)

Region-based methods have become increasingly popular for model-based, monocular 3D tracking of texture-less objects in cluttered scenes. However, while they achieve state-of-the-art results, most methods are computationally expensive, requiring significant resources to run in real-time. In the following, we build on our previous work and develop SRT3D, a sparse region-based approach to 3D object tracking that bridges this gap in efficiency. Our method considers image information sparsely along so-called correspondence lines that model the probability of the object's contour location. We thereby improve on the current state of the art and introduce smoothed step functions that consider a defined global and local uncertainty. For the resulting probabilistic formulation, a thorough analysis is provided. Finally, we use a pre-rendered sparse viewpoint model to create a joint posterior probability for the object pose. The function is maximized using second-order Newton optimization with Tikhonov regularization. During the pose estimation, we differentiate between global and local optimization, using a novel approximation for the first-order derivative employed in the Newton method. In multiple experiments, we demonstrate that the resulting algorithm improves the current state of the art both in terms of runtime and quality, performing particularly well for noisy and cluttered images encountered in the real world.