# Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving

arXiv: https://arxiv.org/abs/2108.04602 · HTML: https://mlsanity.com/paper/2108.04602/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Builds on (most cited first, 13 of 13)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking](https://mlsanity.com/paper/2007.14557/) (2020)
- [RetinaTrack: Online Single Stage Joint Detection and Tracking](https://mlsanity.com/paper/2003.13870/) (2020)
- [Robust Multi-Modality Multi-Object Tracking](https://mlsanity.com/paper/1909.03850/) (2019)
- [JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset](https://mlsanity.com/paper/2002.08397/) (2020)
- [Mono-Camera 3D Multi-Object Tracking Using Deep Learning Detections and PMBM Filtering](https://mlsanity.com/paper/1802.09975/) (2018)
- [End-to-end Learning of Multi-sensor 3D Tracking by Detection](https://mlsanity.com/paper/1806.11534/) (2018)

## Built on by (most cited first, 26 of 26)

- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://mlsanity.com/paper/2202.12100/) (2022)
- [TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM](https://mlsanity.com/paper/2210.16204/) (2022)
- [MotionTrack: End-to-End Transformer-based Multi-Object Tracing with LiDAR-Camera Fusion](https://mlsanity.com/paper/2306.17000/) (2023)
- [Transformers for Multi-Object Tracking on Point Clouds](https://mlsanity.com/paper/2205.15730/) (2022)
- [Fast-Poly: A Fast Polyhedral Framework For 3D Multi-Object Tracking](https://mlsanity.com/paper/2403.13443/) (2024)
- [LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds](https://mlsanity.com/paper/2308.09908/) (2023)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [You Only Need Two Detectors to Achieve Multi-Modal 3D Multi-Object Tracking](https://mlsanity.com/paper/2304.08709/) (2023)
- [3DMODT: Attention-Guided Affinities for Joint Detection & Tracking in 3D Point Clouds](https://mlsanity.com/paper/2211.00746/) (2022)
- [BiTrack: Bidirectional Offline 3D Multi-Object Tracking Using Camera-LiDAR Data](https://mlsanity.com/paper/2406.18414/) (2024)
- [3D Multiple Object Tracking on Autonomous Driving: A Literature Review](https://mlsanity.com/paper/2309.15411/) (2023)
- [3D Multi-Object Tracking Based on Uncertainty-Guided Data Association](https://mlsanity.com/paper/2303.01786/) (2023)
- [Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunities](https://mlsanity.com/paper/2306.03454/) (2023)
- [InScope: A New Real-world 3D Infrastructure-side Collaborative Perception Dataset for Open Traffic Scenarios](https://mlsanity.com/paper/2407.21581/) (2024)
- [CrossTracker: Robust Multi-modal 3D Multi-Object Tracking via Cross Correction](https://mlsanity.com/paper/2411.18850/) (2024)
- [IMM-MOT: A Novel 3D Multi-object Tracking Framework with Interacting Multiple Model Filter](https://mlsanity.com/paper/2502.09672/) (2025)
- [Offline-Poly: A Polyhedral Framework For Offline 3D Multi-Object Tracking](https://mlsanity.com/paper/2602.13772/) (2026)
- [Inverse Neural Rendering for Explainable Multi-Object Tracking](https://mlsanity.com/paper/2404.12359/) (2024)
- [Bayesian Approximation-Based Trajectory Prediction and Tracking with 4D Radar](https://mlsanity.com/paper/2502.01357/) (2025)
- [OptiPMB: Enhancing 3D Multi-Object Tracking with Optimized Poisson Multi-Bernoulli Filtering](https://mlsanity.com/paper/2503.12968/) (2025)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [Polycepta: Object-Centric Appearance Estimation for Multi-Object Tracking](https://mlsanity.com/paper/2606.23604/) (2026)
- [ImLiDAR: Cross-Sensor Dynamic Message Propagation Network for 3D Object Detection](https://mlsanity.com/paper/2211.09518/) (2022)
- [TeFF: Tracking-enhanced Forgetting-free Few-shot 3D LiDAR Semantic Segmentation](https://mlsanity.com/paper/2408.15657/) (2024)
- [RegTrack: Simplicity Beneath Complexity in Robust Multi-Modal 3D Multi-Object Tracking](https://mlsanity.com/paper/2409.00618/) (2024)

## Abstract (arXiv)

Multi-object tracking (MOT) with camera-LiDAR fusion demands accurate results of object detection, affinity computation and data association in real time. This paper presents an efficient multi-modal MOT framework with online joint detection and tracking schemes and robust data association for autonomous driving applications. The novelty of this work includes: (1) development of an end-to-end deep neural network for joint object detection and correlation using 2D and 3D measurements; (2) development of a robust affinity computation module to compute occlusion-aware appearance and motion affinities in 3D space; (3) development of a comprehensive data association module for joint optimization among detection confidences, affinities and start-end probabilities. The experiment results on the KITTI tracking benchmark demonstrate the superior performance of the proposed method in terms of both tracking accuracy and processing speed.