# Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers

arXiv: https://arxiv.org/abs/2103.14829 · HTML: https://mlsanity.com/paper/2103.14829/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 112 references that are in the map.
- First posted on arXiv 27 Mar 2021.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)

## Built on by (most cited first, 21 of 21)

- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [TransCenter: Transformers with Dense Representations for Multiple-Object Tracking](https://mlsanity.com/paper/2103.15145/) (2021)
- [JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Tracking](https://mlsanity.com/paper/2210.11940/) (2022)
- [MotionTrack: End-to-End Transformer-based Multi-Object Tracing with LiDAR-Camera Fusion](https://mlsanity.com/paper/2306.17000/) (2023)
- [Temporal Action Proposal Generation with Transformers](https://mlsanity.com/paper/2105.12043/) (2021)
- [Exploiting More Information in Sparse Point Cloud for 3D Single Object Tracking](https://mlsanity.com/paper/2210.00519/) (2022)
- [Synthesis-based Imaging-Differentiation Representation Learning for Multi-Sequence 3D/4D MRI](https://mlsanity.com/paper/2302.00517/) (2023)
- [Learning Data Association for Multi-Object Tracking using Only Coordinates](https://mlsanity.com/paper/2403.08018/) (2024)
- [MCTR: Multi Camera Tracking Transformer](https://mlsanity.com/paper/2408.13243/) (2024)
- [Contrastive Learning for Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2311.08043/) (2023)
- [PD-SORT: Occlusion-Robust Multi-Object Tracking Using Pseudo-Depth Cues](https://mlsanity.com/paper/2501.11288/) (2025)
- [Deep Learning-Based Robust Multi-Object Tracking via Fusion of mmWave Radar and Camera Sensors](https://mlsanity.com/paper/2407.08049/) (2024)
- [Multi-step Temporal Modeling for UAV Tracking](https://mlsanity.com/paper/2403.04363/) (2024)
- [Bayesian Multiobject Tracking With Neural-Enhanced Motion and Measurement Models](https://mlsanity.com/paper/2506.18124/) (2025)
- [BOTT: Box Only Transformer Tracker for 3D Object Tracking](https://mlsanity.com/paper/2308.08753/) (2023)
- [BASE: Probably a Better Approach to Multi-Object Tracking](https://mlsanity.com/paper/2309.12035/) (2023)
- [Occlusion-Aware SORT: Observing Occlusion for Robust Multi-Object Tracking](https://mlsanity.com/paper/2603.06034/) (2026)
- [TransFiner: A Full-Scale Refinement Approach for Multiple Object Tracking](https://mlsanity.com/paper/2207.12967/) (2022)
- [Mesh-SORT: Simple and effective location-wise tracker with lost management strategies](https://mlsanity.com/paper/2302.14415/) (2023)
- [Tracking by Detection and Query: An Efficient End-to-End Framework for Multi-Object Tracking](https://mlsanity.com/paper/2411.06197/) (2024)
- [Resilient Concurrent Causal Discovery for Topological Event Sequences](https://mlsanity.com/paper/2608.21815/) (2026)

## Abstract (arXiv)

Tracking a time-varying indefinite number of objects in a video sequence over time remains a challenge despite recent advances in the field. Most existing approaches are not able to properly handle multi-object tracking challenges such as occlusion, in part because they ignore long-term temporal information. To address these shortcomings, we present MO3TR: a truly end-to-end Transformer-based online multi-object tracking (MOT) framework that learns to handle occlusions, track initiation and termination without the need for an explicit data association module or any heuristics. MO3TR encodes object interactions into long-term temporal embeddings using a combination of spatial and temporal Transformers, and recursively uses the information jointly with the input data to estimate the states of all tracked objects over time. The spatial attention mechanism enables our framework to learn implicit representations between all the objects and the objects to the measurements, while the temporal attention mechanism focuses on specific parts of past information, allowing our approach to resolve occlusions over multiple frames. Our experiments demonstrate the potential of this new approach, achieving results on par with or better than the current state-of-the-art on multiple MOT metrics for several popular multi-object tracking benchmarks.