# A Multi-body Tracking Framework - From Rigid Objects to Kinematic Structures

arXiv: https://arxiv.org/abs/2208.01502 · HTML: https://mlsanity.com/paper/2208.01502/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 61 references that are in the map.
- First posted on arXiv 2 Aug 2022.

## Builds on (most cited first, 13 of 13)

- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects](https://mlsanity.com/paper/2203.05334/) (2022)
- [Depth-Based Object Tracking Using a Robust Gaussian Filter](https://mlsanity.com/paper/1602.06157/) (2016)
- [SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World](https://mlsanity.com/paper/2110.12715/) (2021)
- [CRAVES: Controlling Robotic Arm with a Vision-based Economic System](https://mlsanity.com/paper/1812.00725/) (2018)
- [Probabilistic Articulated Real-Time Tracking for Robot Manipulation](https://mlsanity.com/paper/1610.04871/) (2016)
- [The Coordinate Particle Filter - A novel Particle Filter for High Dimensional Systems](https://mlsanity.com/paper/1505.00251/) (2015)

## Built on by (most cited first, 2 of 2)

- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)

## Abstract (arXiv)

Kinematic structures are very common in the real world. They range from simple articulated objects to complex mechanical systems. However, despite their relevance, most model-based 3D tracking methods only consider rigid objects. To overcome this limitation, we propose a flexible framework that allows the extension of existing 6DoF algorithms to kinematic structures. Our approach focuses on methods that employ Newton-like optimization techniques, which are widely used in object tracking. The framework considers both tree-like and closed kinematic structures and allows a flexible configuration of joints and constraints. To project equations from individual rigid bodies to a multi-body system, Jacobians are used. For closed kinematic chains, a novel formulation that features Lagrange multipliers is developed. In a detailed mathematical proof, we show that our constraint formulation leads to an exact kinematic solution and converges in a single iteration. Based on the proposed framework, we extend ICG, which is a state-of-the-art rigid object tracking algorithm, to multi-body tracking. For the evaluation, we create a highly-realistic synthetic dataset that features a large number of sequences and various robots. Based on this dataset, we conduct a wide variety of experiments that demonstrate the excellent performance of the developed framework and our multi-body tracker.