# Multi-Object Tracking with Deep Learning Ensemble for Unmanned Aerial System Applications

arXiv: https://arxiv.org/abs/2110.02044 · HTML: https://mlsanity.com/paper/2110.02044/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking](https://mlsanity.com/paper/1804.00518/) (2018)
- [Detect to Track and Track to Detect](https://mlsanity.com/paper/1710.03958/) (2017)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [Unsupervised Pre-training for Person Re-identification](https://mlsanity.com/paper/2012.03753/) (2020)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification](https://mlsanity.com/paper/1810.02936/) (2018)
- [How To Train Your Deep Multi-Object Tracker](https://mlsanity.com/paper/1906.06618/) (2019)
- [On the Unreasonable Effectiveness of Centroids in Image Retrieval](https://mlsanity.com/paper/2104.13643/) (2021)
- [Long Short-Term Memory Kalman Filters:Recurrent Neural Estimators for Pose Regularization](https://mlsanity.com/paper/1708.01885/) (2017)
- [Learning Shape Representations for Clothing Variations in Person Re-Identification](https://mlsanity.com/paper/2003.07340/) (2020)
- [Deep Continuous Conditional Random Fields with Asymmetric Inter-object Constraints for Online Multi-object Tracking](https://mlsanity.com/paper/1806.01183/) (2018)
- [A Hybrid Data Association Framework for Robust Online Multi-Object Tracking](https://mlsanity.com/paper/1703.10764/) (2017)

## Abstract (arXiv)

Multi-object tracking (MOT) is a crucial component of situational awareness in military defense applications. With the growing use of unmanned aerial systems (UASs), MOT methods for aerial surveillance is in high demand. Application of MOT in UAS presents specific challenges such as moving sensor, changing zoom levels, dynamic background, illumination changes, obscurations and small objects. In this work, we present a robust object tracking architecture aimed to accommodate for the noise in real-time situations. We propose a kinematic prediction model, called Deep Extended Kalman Filter (DeepEKF), in which a sequence-to-sequence architecture is used to predict entity trajectories in latent space. DeepEKF utilizes a learned image embedding along with an attention mechanism trained to weight the importance of areas in an image to predict future states. For the visual scoring, we experiment with different similarity measures to calculate distance based on entity appearances, including a convolutional neural network (CNN) encoder, pre-trained using Siamese networks. In initial evaluation experiments, we show that our method, combining scoring structure of the kinematic and visual models within a MHT framework, has improved performance especially in edge cases where entity motion is unpredictable, or the data presents frames with significant gaps.