# Unsupervised Multiple-Object Tracking with a Dynamical Variational Autoencoder

arXiv: https://arxiv.org/abs/2202.09315 · HTML: https://mlsanity.com/paper/2202.09315/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 18 Feb 2022.

## Builds on (most cited first, 30 of 32)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)
- [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [Learning by tracking: Siamese CNN for robust target association](https://mlsanity.com/paper/1604.07866/) (2016)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [Auxiliary Deep Generative Models](https://mlsanity.com/paper/1602.05473/) (2016)
- [Online Multi-Target Tracking Using Recurrent Neural Networks](https://mlsanity.com/paper/1604.03635/) (2016)
- [Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism](https://mlsanity.com/paper/1708.02843/) (2017)
- [Recurrent Autoregressive Networks for Online Multi-Object Tracking](https://mlsanity.com/paper/1711.02741/) (2017)
- [Learning Deep Feature Representations with Domain Guided Dropout for Person Re-identification](https://mlsanity.com/paper/1604.07528/) (2016)
- [Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking](https://mlsanity.com/paper/1911.09033/) (2019)
- [A Recurrent Variational Autoencoder for Speech Enhancement](https://mlsanity.com/paper/1910.10942/) (2019)
- [Probabilistic Tracklet Scoring and Inpainting for Multiple Object Tracking](https://mlsanity.com/paper/2012.02337/) (2020)
- [UnOVOST: Unsupervised Offline Video Object Segmentation and Tracking](https://mlsanity.com/paper/2001.05425/) (2020)
- [Simple Unsupervised Multi-Object Tracking](https://mlsanity.com/paper/2006.02609/) (2020)
- [Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers](https://mlsanity.com/paper/1809.03137/) (2018)

## Abstract (arXiv)

In this paper, we present an unsupervised probabilistic model and associated estimation algorithm for multi-object tracking (MOT) based on a dynamical variational autoencoder (DVAE), called DVAE-UMOT. The DVAE is a latent-variable deep generative model that can be seen as an extension of the variational autoencoder for the modeling of temporal sequences. It is included in DVAE-UMOT to model the objects' dynamics, after being pre-trained on an unlabeled synthetic dataset of single-object trajectories. Then the distributions and parameters of DVAE-UMOT are estimated on each multi-object sequence to track using the principles of variational inference: Definition of an approximate posterior distribution of the latent variables and maximization of the corresponding evidence lower bound of the data likehood function. DVAE-UMOT is shown experimentally to compete well with and even surpass the performance of two state-of-the-art probabilistic MOT models. Code and data are publicly available.