# Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking

arXiv: https://arxiv.org/abs/2101.12159 · HTML: https://mlsanity.com/paper/2101.12159/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification](https://mlsanity.com/paper/1809.04427/) (2018)
- [Online Multi-Object Tracking with Dual Matching Attention Networks](https://mlsanity.com/paper/1902.00749/) (2019)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [Spatial-Temporal Relation Networks for Multi-Object Tracking](https://mlsanity.com/paper/1904.11489/) (2019)
- [How To Train Your Deep Multi-Object Tracker](https://mlsanity.com/paper/1906.06618/) (2019)
- [FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking](https://mlsanity.com/paper/1904.04989/) (2019)
- [Online Multi-Target Tracking Using Recurrent Neural Networks](https://mlsanity.com/paper/1604.03635/) (2016)
- [Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism](https://mlsanity.com/paper/1708.02843/) (2017)
- [Recurrent Autoregressive Networks for Online Multi-Object Tracking](https://mlsanity.com/paper/1711.02741/) (2017)
- [Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor](https://mlsanity.com/paper/1504.02340/) (2015)
- [CVPR19 Tracking and Detection Challenge: How crowded can it get?](https://mlsanity.com/paper/1906.04567/) (2019)
- [Online Multi-Object Tracking with Instance-Aware Tracker and Dynamic Model Refreshment](https://mlsanity.com/paper/1902.08231/) (2019)
- [Online Multi-Object Tracking with Historical Appearance Matching and Scene Adaptive Detection Filtering](https://mlsanity.com/paper/1805.10916/) (2018)

## Built on by (most cited first, 22 of 22)

- [Simple Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2206.04656/) (2022)
- [TransCenter: Transformers with Dense Representations for Multiple-Object Tracking](https://mlsanity.com/paper/2103.15145/) (2021)
- [A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds](https://mlsanity.com/paper/2203.04232/) (2022)
- [Towards Generalizable Multi-Object Tracking](https://mlsanity.com/paper/2406.00429/) (2024)
- [Online Multi-Object Tracking with Unsupervised Re-Identification Learning and Occlusion Estimation](https://mlsanity.com/paper/2201.01297/) (2022)
- [In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems](https://mlsanity.com/paper/2209.04796/) (2022)
- [Multi-view Tracking Using Weakly Supervised Human Motion Prediction](https://mlsanity.com/paper/2210.10771/) (2022)
- [Multi-Object Tracking and Segmentation via Neural Message Passing](https://mlsanity.com/paper/2207.07454/) (2022)
- [Trajectory Forecasting from Detection with Uncertainty-Aware Motion Encoding](https://mlsanity.com/paper/2202.01478/) (2022)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Weakly Supervised Video Individual CountingWeakly Supervised Video Individual Counting](https://mlsanity.com/paper/2312.05923/) (2023)
- [DeepScale: Online Frame Size Adaptation for Multi-object Tracking on Smart Cameras and Edge Servers](https://mlsanity.com/paper/2107.10404/) (2021)
- [Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking](https://mlsanity.com/paper/2205.15495/) (2022)
- [Learnable Graph Matching: A Practical Paradigm for Data Association](https://mlsanity.com/paper/2303.15414/) (2023)
- [Sparse3DTrack: Monocular 3D Object Tracking Using Sparse Supervision](https://mlsanity.com/paper/2603.18298/) (2026)
- [Transformer-based assignment decision network for multiple object tracking](https://mlsanity.com/paper/2208.03571/) (2022)
- [STURE: Spatial-Temporal Mutual Representation Learning for Robust Data Association in Online Multi-Object Tracking](https://mlsanity.com/paper/2201.06824/) (2022)
- [Sparse Message Passing Network with Feature Integration for Online Multiple Object Tracking](https://mlsanity.com/paper/2212.02992/) (2022)
- [Real-time Multi-Object Tracking Based on Bi-directional Matching](https://mlsanity.com/paper/2303.08444/) (2023)
- [FACT: Feature Adaptive Continual-learning Tracker for Multiple Object Tracking](https://mlsanity.com/paper/2409.07904/) (2024)
- [SAM2Auto: Auto Annotation Using FLASH](https://mlsanity.com/paper/2506.07850/) (2025)
- [SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos](https://mlsanity.com/paper/2508.14607/) (2025)

## Abstract (arXiv)

In multi-object tracking, the tracker maintains in its memory the appearance and motion information for each object in the scene. This memory is utilized for finding matches between tracks and detections and is updated based on the matching result. Many approaches model each target in isolation and lack the ability to use all the targets in the scene to jointly update the memory. This can be problematic when there are similar looking objects in the scene. In this paper, we solve the problem of simultaneously considering all tracks during memory updating, with only a small spatial overhead, via a novel multi-track pooling module. We additionally propose a training strategy adapted to multi-track pooling which generates hard tracking episodes online. We show that the combination of these innovations results in a strong discriminative appearance model, enabling the use of greedy data association to achieve online tracking performance. Our experiments demonstrate real-time, state-of-the-art performance on public multi-object tracking (MOT) datasets.