# Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?

arXiv: https://arxiv.org/abs/2210.07681 · HTML: https://mlsanity.com/paper/2210.07681/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 83 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)

## Built on by (most cited first, 29 of 29)

- [MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2307.15700/) (2023)
- [MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking](https://mlsanity.com/paper/2303.10404/) (2023)
- [Unifying Short and Long-Term Tracking with Graph Hierarchies](https://mlsanity.com/paper/2212.03038/) (2022)
- [Multiple Object Tracking as ID Prediction](https://mlsanity.com/paper/2403.16848/) (2024)
- [Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking](https://mlsanity.com/paper/2302.03802/) (2023)
- [MotionTrack: Learning Motion Predictor for Multiple Object Tracking](https://mlsanity.com/paper/2306.02585/) (2023)
- [BoostTrack++: using tracklet information to detect more objects in multiple object tracking](https://mlsanity.com/paper/2408.13003/) (2024)
- [TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses](https://mlsanity.com/paper/2306.05888/) (2023)
- [TrackFlow: Multi-Object Tracking with Normalizing Flows](https://mlsanity.com/paper/2308.11513/) (2023)
- [DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusions](https://mlsanity.com/paper/2309.04682/) (2023)
- [DeconfuseTrack:Dealing with Confusion for Multi-Object Tracking](https://mlsanity.com/paper/2403.02767/) (2024)
- [Objects do not disappear: Video object detection by single-frame object location anticipation](https://mlsanity.com/paper/2308.04770/) (2023)
- [PD-SORT: Occlusion-Robust Multi-Object Tracking Using Pseudo-Depth Cues](https://mlsanity.com/paper/2501.11288/) (2025)
- [DepthMOT: Depth Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2404.05518/) (2024)
- [CAMOT: Camera Angle-aware Multi-Object Tracking](https://mlsanity.com/paper/2409.17533/) (2024)
- [OccluTrack: Rethinking Awareness of Occlusion for Enhancing Multiple Pedestrian Tracking](https://mlsanity.com/paper/2309.10360/) (2023)
- [NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking](https://mlsanity.com/paper/2509.02111/) (2025)
- [Tracking Objects with 3D Representation from Videos](https://mlsanity.com/paper/2306.05416/) (2023)
- [Diffusion-Based Particle-DETR for BEV Perception](https://mlsanity.com/paper/2312.11578/) (2023)
- [Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes](https://mlsanity.com/paper/2405.20743/) (2024)
- [History-Aware Transformation of ReID Features for Multiple Object Tracking](https://mlsanity.com/paper/2503.12562/) (2025)
- [UTOPIA: Unconstrained Tracking Objects without Preliminary Examination via Cross-Domain Adaptation](https://mlsanity.com/paper/2306.09613/) (2023)
- [DINTR: Tracking via Diffusion-based Interpolation](https://mlsanity.com/paper/2410.10053/) (2024)
- [GRASPTrack: Geometry-Reasoned Association via Segmentation and Projection for Multi-Object Tracking](https://mlsanity.com/paper/2508.08117/) (2025)
- [An Improved End-to-End Multi-Target Tracking Method Based on Transformer Self-Attention](https://mlsanity.com/paper/2211.06001/) (2022)
- [Trajectory Prediction in Dynamic Object Tracking: A Critical Study](https://mlsanity.com/paper/2506.19341/) (2025)
- [SAMOFT: Robust Multi-Object Tracking via Region and Flow](https://mlsanity.com/paper/2605.09417/) (2026)
- [Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking](https://mlsanity.com/paper/2609.00924/) (2026)
- [SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning](https://mlsanity.com/paper/2610.14439/) (2026)

## Abstract (arXiv)

Recent developments in monocular multi-object tracking have been very successful in tracking visible objects and bridging short occlusion gaps, mainly relying on data-driven appearance models. While we have significantly advanced short-term tracking performance, bridging longer occlusion gaps remains elusive: state-of-the-art object trackers only bridge less than 10% of occlusions longer than three seconds. We suggest that the missing key is reasoning about future trajectories over a longer time horizon. Intuitively, the longer the occlusion gap, the larger the search space for possible associations. In this paper, we show that even a small yet diverse set of trajectory predictions for moving agents will significantly reduce this search space and thus improve long-term tracking robustness. Our experiments suggest that the crucial components of our approach are reasoning in a bird's-eye view space and generating a small yet diverse set of forecasts while accounting for their localization uncertainty. This way, we can advance state-of-the-art trackers on the MOTChallenge dataset and significantly improve their long-term tracking performance. This paper's source code and experimental data are available at https://github.com/dendorferpatrick/QuoVadis.