# You Only Need Two Detectors to Achieve Multi-Modal 3D Multi-Object Tracking arXiv: https://arxiv.org/abs/2304.08709 ยท HTML: https://mlsanity.com/paper/2304.08709/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 45 of its 54 references that are in the map. - First posted on arXiv 18 Apr 2023. ## Builds on (most cited first, 30 of 45) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021) - [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018) - [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017) - [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019) - [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016) - [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021) - [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019) - [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020) - [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016) - [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021) - [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020) - [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020) - [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020) - [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019) - [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020) - [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020) - [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022) - [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020) - [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019) - [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020) - [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022) - [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020) - [TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking](https://mlsanity.com/paper/2104.00194/) (2021) - [Toward Transformer-Based Object Detection](https://mlsanity.com/paper/2012.09958/) (2020) ## Built on by (most cited first, 8 of 8) - [MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving](https://mlsanity.com/paper/2409.16149/) (2024) - [Survey on video anomaly detection in dynamic scenes with moving cameras](https://mlsanity.com/paper/2308.07050/) (2023) - [3D Multiple Object Tracking on Autonomous Driving: A Literature Review](https://mlsanity.com/paper/2309.15411/) (2023) - [StreamMOTP: Streaming and Unified Framework for Joint 3D Multi-Object Tracking and Trajectory Prediction](https://mlsanity.com/paper/2406.19844/) (2024) - [CrossTracker: Robust Multi-modal 3D Multi-Object Tracking via Cross Correction](https://mlsanity.com/paper/2411.18850/) (2024) - [LiDAR MOT-DETR: A LiDAR-based Two-Stage Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2505.12753/) (2025) - [One Homography is All You Need: IMM-based Joint Homography and Multiple Object State Estimation](https://mlsanity.com/paper/2409.02562/) (2024) - [ODTFormer: Efficient Obstacle Detection and Tracking with Stereo Cameras Based on Transformer](https://mlsanity.com/paper/2403.14626/) (2024) ## Abstract (arXiv) In the classical tracking-by-detection (TBD) paradigm, detection and tracking are separately and sequentially conducted, and data association must be properly performed to achieve satisfactory tracking performance. In this paper, a new end-to-end multi-object tracking framework is proposed, which integrates object detection and multi-object tracking into a single model. The proposed tracking framework eliminates the complex data association process in the classical TBD paradigm, and requires no additional training. Secondly, the regression confidence of historical trajectories is investigated, and the possible states of a trajectory (weak object or strong object) in the current frame are predicted. Then, a confidence fusion module is designed to guide non-maximum suppression for trajectories and detections to achieve ordered and robust tracking. Thirdly, by integrating historical trajectory features, the regression performance of the detector is enhanced, which better reflects the occlusion and disappearance patterns of objects in real world. Lastly, extensive experiments are conducted on the commonly used KITTI and Waymo datasets. The results show that the proposed framework can achieve robust tracking by using only a 2D detector and a 3D detector, and it is proven more accurate than many of the state-of-the-art TBD-based multi-modal tracking methods. The source codes of the proposed method are available at https://github.com/wangxiyang2022/YONTD-MOT.