# 3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking

arXiv: https://arxiv.org/abs/2308.06635 · HTML: https://mlsanity.com/paper/2308.06635/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [Unifying Voxel-based Representation with Transformer for 3D Object Detection](https://mlsanity.com/paper/2206.00630/) (2022)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking](https://mlsanity.com/paper/2104.00194/) (2021)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2012.13755/) (2020)

## Built on by (most cited first, 21 of 21)

- [Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception](https://mlsanity.com/paper/2403.07746/) (2024)
- [MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving](https://mlsanity.com/paper/2409.16149/) (2024)
- [Towards Category Unification of 3D Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2401.11204/) (2024)
- [Fast-Poly: A Fast Polyhedral Framework For 3D Multi-Object Tracking](https://mlsanity.com/paper/2403.13443/) (2024)
- [LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds](https://mlsanity.com/paper/2308.09908/) (2023)
- [Offline Tracking with Object Permanence](https://mlsanity.com/paper/2310.01288/) (2023)
- [Tracking Meets Large Multimodal Models for Driving Scenario Understanding](https://mlsanity.com/paper/2503.14498/) (2025)
- [Zero-Shot 4D Lidar Panoptic Segmentation](https://mlsanity.com/paper/2504.00848/) (2025)
- [TruckDrive: Long-Range Autonomous Highway Driving Dataset](https://mlsanity.com/paper/2603.02413/) (2026)
- [3D Multiple Object Tracking on Autonomous Driving: A Literature Review](https://mlsanity.com/paper/2309.15411/) (2023)
- [3D Single-object Tracking in Point Clouds with High Temporal Variation](https://mlsanity.com/paper/2408.02049/) (2024)
- [Street Gaussians without 3D Object Tracker](https://mlsanity.com/paper/2412.05548/) (2024)
- [LiDAR MOT-DETR: A LiDAR-based Two-Stage Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2505.12753/) (2025)
- [Learning better representations for crowded pedestrians in offboard LiDAR-camera 3D tracking-by-detection](https://mlsanity.com/paper/2505.16029/) (2025)
- [From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving](https://mlsanity.com/paper/2509.23641/) (2025)
- [OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking](https://mlsanity.com/paper/2407.14047/) (2024)
- [Easy-Poly: An Easy Polyhedral Framework For 3D Multi-Object Tracking](https://mlsanity.com/paper/2502.17822/) (2025)
- [OptiPMB: Enhancing 3D Multi-Object Tracking with Optimized Poisson Multi-Bernoulli Filtering](https://mlsanity.com/paper/2503.12968/) (2025)
- [An Efficient LiDAR-Camera Fusion Network for Multi-Class 3D Dynamic Object Detection and Trajectory Prediction](https://mlsanity.com/paper/2504.13647/) (2025)
- [S3MOT: Monocular 3D Object Tracking with Selective State Space Model](https://mlsanity.com/paper/2504.18068/) (2025)
- [FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2510.19981/) (2025)

## Abstract (arXiv)

Tracking 3D objects accurately and consistently is crucial for autonomous vehicles, enabling more reliable downstream tasks such as trajectory prediction and motion planning. Based on the substantial progress in object detection in recent years, the tracking-by-detection paradigm has become a popular choice due to its simplicity and efficiency. State-of-the-art 3D multi-object tracking (MOT) approaches typically rely on non-learned model-based algorithms such as Kalman Filter but require many manually tuned parameters. On the other hand, learning-based approaches face the problem of adapting the training to the online setting, leading to inevitable distribution mismatch between training and inference as well as suboptimal performance. In this work, we propose 3DMOTFormer, a learned geometry-based 3D MOT framework building upon the transformer architecture. We use an Edge-Augmented Graph Transformer to reason on the track-detection bipartite graph frame-by-frame and conduct data association via edge classification. To reduce the distribution mismatch between training and inference, we propose a novel online training strategy with an autoregressive and recurrent forward pass as well as sequential batch optimization. Using CenterPoint detections, our approach achieves 71.2% and 68.2% AMOTA on the nuScenes validation and test split, respectively. In addition, a trained 3DMOTFormer model generalizes well across different object detectors. Code is available at: https://github.com/dsx0511/3DMOTFormer.