# Next Generation Multitarget Trackers: Random Finite Set Methods vs Transformer-based Deep Learning

arXiv: https://arxiv.org/abs/2104.00734 · HTML: https://mlsanity.com/paper/2104.00734/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation](https://mlsanity.com/paper/2011.00747/) (2020)
- [Generalized optimal sub-pattern assignment metric](https://mlsanity.com/paper/1601.05585/) (2016)
- [Poisson multi-Bernoulli mixture filter: direct derivation and implementation](https://mlsanity.com/paper/1703.04264/) (2017)
- [No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles using Cameras & LiDARs](https://mlsanity.com/paper/1802.08755/) (2018)
- [Gaussian implementation of the multi-Bernoulli mixture filter](https://mlsanity.com/paper/1908.08819/) (2019)

## Built on by (most cited first, 6 of 6)

- [Can Deep Learning be Applied to Model-Based Multi-Object Tracking?](https://mlsanity.com/paper/2202.07909/) (2022)
- [Multi-Target Tracking with Transferable Convolutional Neural Networks](https://mlsanity.com/paper/2210.15539/) (2022)
- [Transformer-Based Multi-Object Smoothing with Decoupled Data Association and Smoothing](https://mlsanity.com/paper/2312.17261/) (2023)
- [Deep Fusion of Multi-Object Densities Using Transformer](https://mlsanity.com/paper/2209.08857/) (2022)
- [Integrated Detection and Tracking Based on Radar Range-Doppler Feature](https://mlsanity.com/paper/2509.06569/) (2025)
- [Causal neural set filtering for online multi-target tracking](https://mlsanity.com/paper/2609.16054/) (2026)

## Abstract (arXiv)

Multitarget Tracking (MTT) is the problem of tracking the states of an unknown number of objects using noisy measurements, with important applications to autonomous driving, surveillance, robotics, and others. In the model-based Bayesian setting, there are conjugate priors that enable us to express the multi-object posterior in closed form, which could theoretically provide Bayes-optimal estimates. However, the posterior involves a super-exponential growth of the number of hypotheses over time, forcing state-of-the-art methods to resort to approximations for remaining tractable, which can impact their performance in complex scenarios. Model-free methods based on deep-learning provide an attractive alternative, as they can, in principle, learn the optimal filter from data, but to the best of our knowledge were never compared to current state-of-the-art Bayesian filters, specially not in contexts where accurate models are available. In this paper, we propose a high-performing deep-learning method for MTT based on the Transformer architecture and compare it to two state-of-the-art Bayesian filters, in a setting where we assume the correct model is provided. Although this gives an edge to the model-based filters, it also allows us to generate unlimited training data. We show that the proposed model outperforms state-of-the-art Bayesian filters in complex scenarios, while matching their performance in simpler cases, which validates the applicability of deep-learning also in the model-based regime. The code for all our implementations is made available at https://github.com/JulianoLagana/MT3 .