# CrossTracker: Robust Multi-modal 3D Multi-Object Tracking via Cross Correction

arXiv: https://arxiv.org/abs/2411.18850 · HTML: https://mlsanity.com/paper/2411.18850/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 68 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Virtual Sparse Convolution for Multimodal 3D Object Detection](https://mlsanity.com/paper/2303.02314/) (2023)
- [FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking](https://mlsanity.com/paper/1904.04989/) (2019)
- [AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics](https://mlsanity.com/paper/2008.08063/) (2020)
- [Robust Multi-Modality Multi-Object Tracking](https://mlsanity.com/paper/1909.03850/) (2019)
- [DEFT: Detection Embeddings for Tracking](https://mlsanity.com/paper/2102.02267/) (2021)
- [Track to Reconstruct and Reconstruct to Track](https://mlsanity.com/paper/1910.00130/) (2019)
- [Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor](https://mlsanity.com/paper/1504.02340/) (2015)
- [Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking](https://mlsanity.com/paper/1802.09298/) (2018)
- [JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset](https://mlsanity.com/paper/2002.08397/) (2020)
- [Optimization-Based Control for Dynamic Legged Robots](https://mlsanity.com/paper/2211.11644/) (2022)

## Built on by (most cited first, 2 of 2)

- [PVNet: Point-Voxel Interaction LiDAR Scene Upsampling Via Diffusion Models](https://mlsanity.com/paper/2508.17050/) (2025)
- [MotionSync: Non-Causal Refinement of Causal Tracker for Label-Efficient 3D Perception](https://mlsanity.com/paper/2608.29567/) (2026)

## Abstract (arXiv)

The fusion of camera- and LiDAR-based detections offers a promising solution to mitigate tracking failures in 3D multi-object tracking (MOT). However, existing methods predominantly exploit camera detections to correct tracking failures caused by potential LiDAR detection problems, neglecting the reciprocal benefit of refining camera detections using LiDAR data. This limitation is rooted in their single-stage architecture, akin to single-stage object detectors, lacking a dedicated trajectory refinement module to fully exploit the complementary multi-modal information. To this end, we introduce CrossTracker, a novel two-stage paradigm for online multi-modal 3D MOT. CrossTracker operates in a coarse-to-fine manner, initially generating coarse trajectories and subsequently refining them through an independent refinement process. Specifically, CrossTracker incorporates three essential modules: i) a multi-modal modeling (M^3) module that, by fusing multi-modal information (images, point clouds, and even plane geometry extracted from images), provides a robust metric for subsequent trajectory generation. ii) a coarse trajectory generation (C-TG) module that generates initial coarse dual-stream trajectories, and iii) a trajectory refinement (TR) module that refines coarse trajectories through cross correction between camera and LiDAR streams. Comprehensive experiments demonstrate the superior performance of our CrossTracker over its eighteen competitors, underscoring its effectiveness in harnessing the synergistic benefits of camera and LiDAR sensors for robust multi-modal 3D MOT.