# MotionTrack: End-to-End Transformer-based Multi-Object Tracing with LiDAR-Camera Fusion

arXiv: https://arxiv.org/abs/2306.17000 · HTML: https://mlsanity.com/paper/2306.17000/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 53 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries](https://mlsanity.com/paper/2205.00613/) (2022)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving](https://mlsanity.com/paper/2109.06446/) (2021)
- [AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics](https://mlsanity.com/paper/2008.08063/) (2020)
- [Exploring Simple 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2108.10312/) (2021)
- [Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking](https://mlsanity.com/paper/2302.03802/) (2023)
- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [Deep Network Flow for Multi-Object Tracking](https://mlsanity.com/paper/1706.08482/) (2017)

## Built on by (most cited first, 19 of 19)

- [Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge](https://mlsanity.com/paper/2405.16464/) (2024)
- [Advancing Chart Question Answering with Robust Chart Component Recognition](https://mlsanity.com/paper/2407.21038/) (2024)
- [3D Multiple Object Tracking on Autonomous Driving: A Literature Review](https://mlsanity.com/paper/2309.15411/) (2023)
- [JDT3D: Addressing the Gaps in LiDAR-Based Tracking-by-Attention](https://mlsanity.com/paper/2407.04926/) (2024)
- [LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation](https://mlsanity.com/paper/2411.14927/) (2024)
- [Image-Guided Outdoor LiDAR Perception Quality Assessment for Autonomous Driving](https://mlsanity.com/paper/2406.17265/) (2024)
- [LiDAR MOT-DETR: A LiDAR-based Two-Stage Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2505.12753/) (2025)
- [Radar Tracker: Moving Instance Tracking in Sparse and Noisy Radar Point Clouds](https://mlsanity.com/paper/2507.03441/) (2025)
- [nuScenes Revisited: Progress and Challenges in Autonomous Driving](https://mlsanity.com/paper/2512.02448/) (2025)
- [Offline-Poly: A Polyhedral Framework For Offline 3D Multi-Object Tracking](https://mlsanity.com/paper/2602.13772/) (2026)
- [SCATR: Mitigating New Instance Suppression in LiDAR-based Tracking-by-Attention via Second Chance Assignment and Track Query Dropout](https://mlsanity.com/paper/2603.01485/) (2026)
- [OptiPMB: Enhancing 3D Multi-Object Tracking with Optimized Poisson Multi-Bernoulli Filtering](https://mlsanity.com/paper/2503.12968/) (2025)
- [Fusion-Poly: A Polyhedral Framework Based on Spatial-Temporal Fusion for 3D Multi-Object Tracking](https://mlsanity.com/paper/2603.08199/) (2026)
- [Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning](https://mlsanity.com/paper/2603.24257/) (2026)
- [Radar-Informed 3D Multi-Object Tracking under Adverse Conditions](https://mlsanity.com/paper/2604.13571/) (2026)
- [FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2510.19981/) (2025)
- [End-to-End 3-D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration](https://mlsanity.com/paper/2512.21831/) (2025)
- [GateMOT: Q-Gated Attention for Dense Object Tracking](https://mlsanity.com/paper/2604.26353/) (2026)
- [Scene Reconstruction as Mapping Priors for 3D Detection](https://mlsanity.com/paper/2605.22997/) (2026)

## Abstract (arXiv)

Multiple Object Tracking (MOT) is crucial to autonomous vehicle perception. End-to-end transformer-based algorithms, which detect and track objects simultaneously, show great potential for the MOT task. However, most existing methods focus on image-based tracking with a single object category. In this paper, we propose an end-to-end transformer-based MOT algorithm (MotionTrack) with multi-modality sensor inputs to track objects with multiple classes. Our objective is to establish a transformer baseline for the MOT in an autonomous driving environment. The proposed algorithm consists of a transformer-based data association (DA) module and a transformer-based query enhancement module to achieve MOT and Multiple Object Detection (MOD) simultaneously. The MotionTrack and its variations achieve better results (AMOTA score at 0.55) on the nuScenes dataset compared with other classical baseline models, such as the AB3DMOT, the CenterTrack, and the probabilistic 3D Kalman filter. In addition, we prove that a modified attention mechanism can be utilized for DA to accomplish the MOT, and aggregate history features to enhance the MOD performance.