# Large-displacement 3D Object Tracking with Hybrid Non-local Optimization

arXiv: https://arxiv.org/abs/2207.12620 · HTML: https://mlsanity.com/paper/2207.12620/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 11 of 11)

- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [A Region-based Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking](https://mlsanity.com/paper/1807.02087/) (2018)
- [SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World](https://mlsanity.com/paper/2110.12715/) (2021)
- [ROSEFusion: Random Optimization for Online Dense Reconstruction under Fast Camera Motion](https://mlsanity.com/paper/2105.05600/) (2021)

## Built on by (most cited first, 7 of 7)

- [GoTrack: Generic 6DoF Object Pose Refinement and Tracking](https://mlsanity.com/paper/2506.07155/) (2025)
- [For A More Comprehensive Evaluation of 6DoF Object Pose Tracking](https://mlsanity.com/paper/2309.07796/) (2023)
- [Event6D: Event-based Novel Object 6D Pose Tracking](https://mlsanity.com/paper/2603.28045/) (2026)
- [DynaWeightPnP: Toward global real-time 3D-2D solver in PnP without correspondences](https://mlsanity.com/paper/2409.18457/) (2024)
- [Robust 6DoF Pose Tracking Considering Contour and Interior Correspondence Uncertainty for AR Assembly Guidance](https://mlsanity.com/paper/2502.11971/) (2025)
- [Active Control Points-based 6DoF Pose Tracking for Industrial Metal Objects](https://mlsanity.com/paper/2507.01478/) (2025)
- [Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition](https://mlsanity.com/paper/2607.09329/) (2026)

## Abstract (arXiv)

Optimization-based 3D object tracking is known to be precise and fast, but sensitive to large inter-frame displacements. In this paper we propose a fast and effective non-local 3D tracking method. Based on the observation that erroneous local minimum are mostly due to the out-of-plane rotation, we propose a hybrid approach combining non-local and local optimizations for different parameters, resulting in efficient non-local search in the 6D pose space. In addition, a precomputed robust contour-based tracking method is proposed for the pose optimization. By using long search lines with multiple candidate correspondences, it can adapt to different frame displacements without the need of coarse-to-fine search. After the pre-computation, pose updates can be conducted very fast, enabling the non-local optimization to run in real time. Our method outperforms all previous methods for both small and large displacements. For large displacements, the accuracy is greatly improved (81.7% ;v.s.; 19.4%). At the same time, real-time speed (>50fps) can be achieved with only CPU. The source code is available at urlhttps://github.com/cvbubbles/nonlocal-3dtracking.