# Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences

arXiv: https://arxiv.org/abs/2012.08197 · HTML: https://mlsanity.com/paper/2012.08197/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2020.

## Built on by (most cited first, 13 of 13)

- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [CHORE: Contact, Human and Object REconstruction from a single RGB image](https://mlsanity.com/paper/2204.02445/) (2022)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [AutoRF: Learning 3D Object Radiance Fields from Single View Observations](https://mlsanity.com/paper/2204.03593/) (2022)
- [ObjectMatch: Robust Registration using Canonical Object Correspondences](https://mlsanity.com/paper/2212.01985/) (2022)
- [In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems](https://mlsanity.com/paper/2209.04796/) (2022)
- [Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes](https://mlsanity.com/paper/2208.05067/) (2022)
- [Factored Neural Representation for Scene Understanding](https://mlsanity.com/paper/2304.10950/) (2023)
- [CrossTracker: Robust Multi-modal 3D Multi-Object Tracking via Cross Correction](https://mlsanity.com/paper/2411.18850/) (2024)
- [ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization](https://mlsanity.com/paper/2401.08937/) (2024)
- [3D Multi-Object Tracking with Differentiable Pose Estimation](https://mlsanity.com/paper/2206.13785/) (2022)
- [Adversarial Attack for RGB-Event based Visual Object Tracking](https://mlsanity.com/paper/2504.14423/) (2025)

## Abstract (arXiv)

Multi-object tracking from RGB-D video sequences is a challenging problem due to the combination of changing viewpoints, motion, and occlusions over time. We observe that having the complete geometry of objects aids in their tracking, and thus propose to jointly infer the complete geometry of objects as well as track them, for rigidly moving objects over time. Our key insight is that inferring the complete geometry of the objects significantly helps in tracking. By hallucinating unseen regions of objects, we can obtain additional correspondences between the same instance, thus providing robust tracking even under strong change of appearance. From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space. This allows us to derive 6DoF poses for the objects in each frame, along with their correspondence between frames, providing robust object tracking across the RGB-D sequence. Experiments on both synthetic and real-world RGB-D data demonstrate that we achieve state-of-the-art performance on dynamic object tracking. Furthermore, we show that our object completion significantly helps tracking, providing an improvement of 6.5% in mean MOTA.