# Track to Reconstruct and Reconstruct to Track

arXiv: https://arxiv.org/abs/1910.00130 · HTML: https://mlsanity.com/paper/1910.00130/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 30 of 49)

- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking](https://mlsanity.com/paper/2210.06984/) (2022)
- [Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking](https://mlsanity.com/paper/2302.03802/) (2023)
- [OVTrack: Open-Vocabulary Multiple Object Tracking](https://mlsanity.com/paper/2304.08408/) (2023)
- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2312.13308/) (2023)
- [Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation](https://mlsanity.com/paper/2106.11958/) (2021)
- [Dynamic 3D Gaussian Fields for Urban Areas](https://mlsanity.com/paper/2406.03175/) (2024)
- [DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://mlsanity.com/paper/2202.12100/) (2022)
- [CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion](https://mlsanity.com/paper/2212.01247/) (2022)
- [PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking?](https://mlsanity.com/paper/2208.01957/) (2022)
- [Multi-Level Neural Scene Graphs for Dynamic Urban Environments](https://mlsanity.com/paper/2404.00168/) (2024)
- [Track without Appearance: Learn Box and Tracklet Embedding with Local and Global Motion Patterns for Vehicle Tracking](https://mlsanity.com/paper/2108.06029/) (2021)
- [TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM](https://mlsanity.com/paper/2210.16204/) (2022)
- [TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking](https://mlsanity.com/paper/2101.04206/) (2021)
- [CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception](https://mlsanity.com/paper/2507.19239/) (2025)
- [Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration](https://mlsanity.com/paper/2212.07592/) (2022)
- [DG-Labeler and DGL-MOTS Dataset: Boost the Autonomous Driving Perception](https://mlsanity.com/paper/2110.07790/) (2021)
- [Weakly-Supervised Domain Adaptation of Deep Regression Trackers via Reinforced Knowledge Distillation](https://mlsanity.com/paper/2103.14496/) (2021)
- [Multi-Object Tracking and Segmentation via Neural Message Passing](https://mlsanity.com/paper/2207.07454/) (2022)
- [PNAS-MOT: Multi-Modal Object Tracking with Pareto Neural Architecture Search](https://mlsanity.com/paper/2403.15712/) (2024)
- [You Only Need Two Detectors to Achieve Multi-Modal 3D Multi-Object Tracking](https://mlsanity.com/paper/2304.08709/) (2023)
- [PAGE-4D: Disentangled pose and geometry estimation for vggt-4d perception](https://mlsanity.com/paper/2510.17568/) (2025)
- [DirectTracker: 3D Multi-Object Tracking Using Direct Image Alignment and Photometric Bundle Adjustment](https://mlsanity.com/paper/2209.14965/) (2022)
- [UFO-4D: Unposed Feedforward 4D Reconstruction from Two Images](https://mlsanity.com/paper/2602.24290/) (2026)
- [OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning](https://mlsanity.com/paper/2404.03657/) (2024)
- [Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation](https://mlsanity.com/paper/2509.11772/) (2025)

## Abstract (arXiv)

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstruction-based tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.