# SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking

arXiv: https://arxiv.org/abs/2111.09621 · HTML: https://mlsanity.com/paper/2111.09621/

## Key facts

- Cited by 103 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 29 of 29)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Learning by tracking: Siamese CNN for robust target association](https://mlsanity.com/paper/1604.07866/) (2016)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2012.13755/) (2020)
- [How To Train Your Deep Multi-Object Tracker](https://mlsanity.com/paper/1906.06618/) (2019)
- [RetinaTrack: Online Single Stage Joint Detection and Tracking](https://mlsanity.com/paper/2003.13870/) (2020)
- [Probabilistic 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2001.05673/) (2020)
- [Lifted Disjoint Paths with Application in Multiple Object Tracking](https://mlsanity.com/paper/2006.14550/) (2020)
- [Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking](https://mlsanity.com/paper/2103.16178/) (2021)
- [Auto4D: Learning to Label 4D Objects from Sequential Point Clouds](https://mlsanity.com/paper/2101.06586/) (2021)
- [Learnable Online Graph Representations for 3D Multi-Object Tracking](https://mlsanity.com/paper/2104.11747/) (2021)
- [Score refinement for confidence-based 3D multi-object tracking](https://mlsanity.com/paper/2107.04327/) (2021)
- [FANTrack: 3D Multi-Object Tracking with Feature Association Network](https://mlsanity.com/paper/1905.02843/) (2019)
- [Model-free Vehicle Tracking and State Estimation in Point Cloud Sequences](https://mlsanity.com/paper/2103.06028/) (2021)
- [Multi-Person Tracking by Multicut and Deep Matching](https://mlsanity.com/paper/1608.05404/) (2016)
- [Factor Graph based 3D Multi-Object Tracking in Point Clouds](https://mlsanity.com/paper/2008.05309/) (2020)
- [Enhancing the Association in Multi-Object Tracking via Neighbor Graph](https://mlsanity.com/paper/2007.00265/) (2020)

## Built on by (most cited first, 30 of 103)

- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [Visual Point Cloud Forecasting enables Scalable Autonomous Driving](https://mlsanity.com/paper/2312.17655/) (2023)
- [MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries](https://mlsanity.com/paper/2205.00613/) (2022)
- [FocalFormer3D : Focusing on Hard Instance for 3D Object Detection](https://mlsanity.com/paper/2308.04556/) (2023)
- [Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking](https://mlsanity.com/paper/2302.03802/) (2023)
- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving](https://mlsanity.com/paper/2404.06892/) (2024)
- [Immortal Tracker: Tracklet Never Dies](https://mlsanity.com/paper/2111.13672/) (2021)
- [GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training](https://mlsanity.com/paper/2305.08808/) (2023)
- [Dynamic 3D Gaussian Fields for Urban Areas](https://mlsanity.com/paper/2406.03175/) (2024)
- [ADA-Track++: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association](https://mlsanity.com/paper/2405.08909/) (2024)
- [CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion](https://mlsanity.com/paper/2212.01247/) (2022)
- [DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds](https://mlsanity.com/paper/2306.06023/) (2023)
- [Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection](https://mlsanity.com/paper/2304.12315/) (2023)
- [Multi-Level Neural Scene Graphs for Dynamic Urban Environments](https://mlsanity.com/paper/2404.00168/) (2024)
- [TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM](https://mlsanity.com/paper/2210.16204/) (2022)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [SceneTracker: Long-term Scene Flow Estimation Network](https://mlsanity.com/paper/2403.19924/) (2024)
- [DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queries](https://mlsanity.com/paper/2404.00086/) (2024)
- [3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking](https://mlsanity.com/paper/2308.06635/) (2023)
- [RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud](https://mlsanity.com/paper/2309.09737/) (2023)
- [ByteTrackV2: 2D and 3D Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2303.15334/) (2023)
- [MotionTrack: End-to-End Transformer-based Multi-Object Tracing with LiDAR-Camera Fusion](https://mlsanity.com/paper/2306.17000/) (2023)
- [Quality Matters: Embracing Quality Clues for Robust 3D Multi-Object Tracking](https://mlsanity.com/paper/2208.10976/) (2022)
- [MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving](https://mlsanity.com/paper/2409.16149/) (2024)
- [GNN-PMB: A Simple but Effective Online 3D Multi-Object Tracker without Bells and Whistles](https://mlsanity.com/paper/2206.10255/) (2022)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses](https://mlsanity.com/paper/2306.05888/) (2023)

## Abstract (arXiv)

3D multi-object tracking (MOT) has witnessed numerous novel benchmarks and approaches in recent years, especially those under the "tracking-by-detection" paradigm. Despite their progress and usefulness, an in-depth analysis of their strengths and weaknesses is not yet available. In this paper, we summarize current 3D MOT methods into a unified framework by decomposing them into four constituent parts: pre-processing of detection, association, motion model, and life cycle management. We then ascribe the failure cases of existing algorithms to each component and investigate them in detail. Based on the analyses, we propose corresponding improvements which lead to a strong yet simple baseline: SimpleTrack. Comprehensive experimental results on Waymo Open Dataset and nuScenes demonstrate that our final method could achieve new state-of-the-art results with minor modifications. Furthermore, we take additional steps and rethink whether current benchmarks authentically reflect the ability of algorithms for real-world challenges. We delve into the details of existing benchmarks and find some intriguing facts. Finally, we analyze the distribution and causes of remaining failures in name and propose future directions for 3D MOT. Our code is available at https://github.com/TuSimple/SimpleTrack.