# Robust Multi-Modality Multi-Object Tracking

arXiv: https://arxiv.org/abs/1909.03850 · HTML: https://mlsanity.com/paper/1909.03850/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 30 of 56)

- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2108.04728/) (2021)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [DEFT: Detection Embeddings for Tracking](https://mlsanity.com/paper/2102.02267/) (2021)
- [3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds](https://mlsanity.com/paper/2111.04426/) (2021)
- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [3D Siamese Transformer Network for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2207.11995/) (2022)
- [Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2211.02219/) (2022)
- [Learnable Online Graph Representations for 3D Multi-Object Tracking](https://mlsanity.com/paper/2104.11747/) (2021)
- [DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://mlsanity.com/paper/2202.12100/) (2022)
- [3D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attention](https://mlsanity.com/paper/2203.10926/) (2022)
- [Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving](https://mlsanity.com/paper/2108.04602/) (2021)
- [TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM](https://mlsanity.com/paper/2210.16204/) (2022)
- [Self-Supervised Multi-Object Tracking with Cross-Input Consistency](https://mlsanity.com/paper/2111.05943/) (2021)
- [A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds](https://mlsanity.com/paper/2203.04232/) (2022)
- [4D Panoptic Scene Graph Generation](https://mlsanity.com/paper/2405.10305/) (2024)
- [TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking](https://mlsanity.com/paper/2101.04206/) (2021)
- [Self-Supervised Multi-Object Tracking with Path Consistency](https://mlsanity.com/paper/2404.05136/) (2024)
- [Proactive Anomaly Detection for Robot Navigation with Multi-Sensor Fusion](https://mlsanity.com/paper/2204.01146/) (2022)
- [Object-Centric Multiple Object Tracking](https://mlsanity.com/paper/2309.00233/) (2023)
- [MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation](https://mlsanity.com/paper/2110.09481/) (2021)
- [Transformers for Multi-Object Tracking on Point Clouds](https://mlsanity.com/paper/2205.15730/) (2022)
- [TrackFlow: Multi-Object Tracking with Normalizing Flows](https://mlsanity.com/paper/2308.11513/) (2023)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [Fast-Poly: A Fast Polyhedral Framework For 3D Multi-Object Tracking](https://mlsanity.com/paper/2403.13443/) (2024)
- [SpOT: Spatiotemporal Modeling for 3D Object Tracking](https://mlsanity.com/paper/2207.05856/) (2022)
- [LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds](https://mlsanity.com/paper/2308.09908/) (2023)
- [PNAS-MOT: Multi-Modal Object Tracking with Pareto Neural Architecture Search](https://mlsanity.com/paper/2403.15712/) (2024)

## Abstract (arXiv)

Multi-sensor perception is crucial to ensure the reliability and accuracy in autonomous driving system, while multi-object tracking (MOT) improves that by tracing sequential movement of dynamic objects. Most current approaches for multi-sensor multi-object tracking are either lack of reliability by tightly relying on a single input source (e.g., center camera), or not accurate enough by fusing the results from multiple sensors in post processing without fully exploiting the inherent information. In this study, we design a generic sensor-agnostic multi-modality MOT framework (mmMOT), where each modality (i.e., sensors) is capable of performing its role independently to preserve reliability, and further improving its accuracy through a novel multi-modality fusion module. Our mmMOT can be trained in an end-to-end manner, enables joint optimization for the base feature extractor of each modality and an adjacency estimator for cross modality. Our mmMOT also makes the first attempt to encode deep representation of point cloud in data association process in MOT. We conduct extensive experiments to evaluate the effectiveness of the proposed framework on the challenging KITTI benchmark and report state-of-the-art performance. Code and models are available at https://github.com/ZwwWayne/mmMOT.