# InterTrack: Interaction Transformer for 3D Multi-Object Tracking

arXiv: https://arxiv.org/abs/2208.08041 · HTML: https://mlsanity.com/paper/2208.08041/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Universal Correspondence Network](https://mlsanity.com/paper/1606.03558/) (2016)
- [Joint Monocular 3D Vehicle Detection and Tracking](https://mlsanity.com/paper/1811.10742/) (2018)
- [Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2012.13755/) (2020)
- [RetinaTrack: Online Single Stage Joint Detection and Tracking](https://mlsanity.com/paper/2003.13870/) (2020)

## Built on by (most cited first, 6 of 6)

- [TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses](https://mlsanity.com/paper/2306.05888/) (2023)
- [LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds](https://mlsanity.com/paper/2308.09908/) (2023)
- [3D Multiple Object Tracking on Autonomous Driving: A Literature Review](https://mlsanity.com/paper/2309.15411/) (2023)
- [Object Re-Identification from Point Clouds](https://mlsanity.com/paper/2305.10210/) (2023)
- [ShaSTA-Fuse: Camera-LiDAR Sensor Fusion to Model Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking](https://mlsanity.com/paper/2310.02532/) (2023)
- [Multi-Object Tracking based on Imaging Radar 3D Object Detection](https://mlsanity.com/paper/2406.01011/) (2024)

## Abstract (arXiv)

3D multi-object tracking (MOT) is a key problem for autonomous vehicles, required to perform well-informed motion planning in dynamic environments. Particularly for densely occupied scenes, associating existing tracks to new detections remains challenging as existing systems tend to omit critical contextual information. Our proposed solution, InterTrack, introduces the Interaction Transformer for 3D MOT to generate discriminative object representations for data association. We extract state and shape features for each track and detection, and efficiently aggregate global information via attention. We then perform a learned regression on each track/detection feature pair to estimate affinities, and use a robust two-stage data association and track management approach to produce the final tracks. We validate our approach on the nuScenes 3D MOT benchmark, where we observe significant improvements, particularly on classes with small physical sizes and clustered objects. As of submission, InterTrack ranks 1st in overall AMOTA among methods using CenterPoint detections.