# Unsupervised Spatio-temporal Latent Feature Clustering for Multiple-object Tracking and Segmentation

arXiv: https://arxiv.org/abs/2007.07175 · HTML: https://mlsanity.com/paper/2007.07175/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 3 of 3)

- [Mask-Free Video Instance Segmentation](https://mlsanity.com/paper/2303.15904/) (2023)
- [Self-supervised Learning for Panoptic Segmentation of Multiple Fruit Flower Species](https://mlsanity.com/paper/2209.04618/) (2022)
- [Tracking Passengers and Baggage Items using Multiple Overhead Cameras at Security Checkpoints](https://mlsanity.com/paper/2301.00190/) (2022)

## Abstract (arXiv)

Assigning consistent temporal identifiers to multiple moving objects in a video sequence is a challenging problem. A solution to that problem would have immediate ramifications in multiple object tracking and segmentation problems. We propose a strategy that treats the temporal identification task as a spatio-temporal clustering problem. We propose an unsupervised learning approach using a convolutional and fully connected autoencoder, which we call deep heterogeneous autoencoder, to learn discriminative features from segmentation masks and detection bounding boxes. We extract masks and their corresponding bounding boxes from a pretrained instance segmentation network and train the autoencoders jointly using task-dependent uncertainty weights to generate common latent features. We then construct constraints graphs that encourage associations among objects that satisfy a set of known temporal conditions. The feature vectors and the constraints graphs are then provided to the kmeans clustering algorithm to separate the corresponding data points in the latent space. We evaluate the performance of our method using challenging synthetic and real-world multiple-object video datasets. Our results show that our technique outperforms several state-of-the-art methods.