# A Region-based Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking

arXiv: https://arxiv.org/abs/1807.02087 · HTML: https://mlsanity.com/paper/1807.02087/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2018.

## Built on by (most cited first, 21 of 21)

- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects](https://mlsanity.com/paper/2203.05334/) (2022)
- [Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera](https://mlsanity.com/paper/2408.03225/) (2024)
- [SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World](https://mlsanity.com/paper/2110.12715/) (2021)
- [Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks](https://mlsanity.com/paper/2106.14070/) (2021)
- [Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild](https://mlsanity.com/paper/2209.12009/) (2022)
- [PACE: A Large-Scale Dataset with Pose Annotations in Cluttered Environments](https://mlsanity.com/paper/2312.15130/) (2023)
- [GoTrack: Generic 6DoF Object Pose Refinement and Tracking](https://mlsanity.com/paper/2506.07155/) (2025)
- [BCOT: A Markerless High-Precision 3D Object Tracking Benchmark](https://mlsanity.com/paper/2203.13437/) (2022)
- [Large-displacement 3D Object Tracking with Hybrid Non-local Optimization](https://mlsanity.com/paper/2207.12620/) (2022)
- [Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking](https://mlsanity.com/paper/2302.11458/) (2023)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2105.14391/) (2021)
- [DynamicPose: Real-time and Robust 6D Object Pose Tracking for Fast-Moving Cameras and Objects](https://mlsanity.com/paper/2508.11950/) (2025)
- [Combining Local and Global Pose Estimation for Precise Tracking of Similar Objects](https://mlsanity.com/paper/2201.13278/) (2022)
- [For A More Comprehensive Evaluation of 6DoF Object Pose Tracking](https://mlsanity.com/paper/2309.07796/) (2023)
- [Optical Flow-Guided 6DoF Object Pose Tracking with an Event Camera](https://mlsanity.com/paper/2512.21053/) (2025)
- [A Novel Wide-Area Multiobject Detection System with High-Probability Region Searching](https://mlsanity.com/paper/2405.04589/) (2024)
- [Robust 6DoF Pose Tracking Considering Contour and Interior Correspondence Uncertainty for AR Assembly Guidance](https://mlsanity.com/paper/2502.11971/) (2025)
- [Active Control Points-based 6DoF Pose Tracking for Industrial Metal Objects](https://mlsanity.com/paper/2507.01478/) (2025)
- [Robust Visual Localization in Compute-Constrained Environments by Salient Edge Rendering and Weighted Hamming Similarity](https://mlsanity.com/paper/2509.25520/) (2025)

## Abstract (arXiv)

We propose an algorithm for real-time 6DOF pose tracking of rigid 3D objects using a monocular RGB camera. The key idea is to derive a region-based cost function using temporally consistent local color histograms. While such region-based cost functions are commonly optimized using first-order gradient descent techniques, we systematically derive a Gauss-Newton optimization scheme which gives rise to drastically faster convergence and highly accurate and robust tracking performance. We furthermore propose a novel complex dataset dedicated for the task of monocular object pose tracking and make it publicly available to the community. To our knowledge, it is the first to address the common and important scenario in which both the camera as well as the objects are moving simultaneously in cluttered scenes. In numerous experiments - including our own proposed dataset - we demonstrate that the proposed Gauss-Newton approach outperforms existing approaches, in particular in the presence of cluttered backgrounds, heterogeneous objects and partial occlusions.