# Learning by tracking: Siamese CNN for robust target association

arXiv: https://arxiv.org/abs/1604.07866 · HTML: https://mlsanity.com/paper/1604.07866/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2016.

## Built on by (most cited first, 30 of 79)

- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?](https://mlsanity.com/paper/2108.09518/) (2021)
- [4D Panoptic LiDAR Segmentation](https://mlsanity.com/paper/2102.12472/) (2021)
- [SiamMOT: Siamese Multi-Object Tracking](https://mlsanity.com/paper/2105.11595/) (2021)
- [Unifying Short and Long-Term Tracking with Graph Hierarchies](https://mlsanity.com/paper/2212.03038/) (2022)
- [QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking](https://mlsanity.com/paper/2210.06984/) (2022)
- [DEFT: Detection Embeddings for Tracking](https://mlsanity.com/paper/2102.02267/) (2021)
- [OVTrack: Open-Vocabulary Multiple Object Tracking](https://mlsanity.com/paper/2304.08408/) (2023)
- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking](https://mlsanity.com/paper/2103.16178/) (2021)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [Simple Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2206.04656/) (2022)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Tracking Every Thing in the Wild](https://mlsanity.com/paper/2207.12978/) (2022)
- [Learning a Proposal Classifier for Multiple Object Tracking](https://mlsanity.com/paper/2103.07889/) (2021)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [Unified Transformer Tracker for Object Tracking](https://mlsanity.com/paper/2203.15175/) (2022)
- [CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion](https://mlsanity.com/paper/2212.01247/) (2022)
- [Time3D: End-to-End Joint Monocular 3D Object Detection and Tracking for Autonomous Driving](https://mlsanity.com/paper/2205.14882/) (2022)
- [PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking?](https://mlsanity.com/paper/2208.01957/) (2022)
- [Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?](https://mlsanity.com/paper/2210.07681/) (2022)
- [Learning from Demonstration in the Wild](https://mlsanity.com/paper/1811.03516/) (2018)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.14829/) (2021)
- [MESA: Matching Everything by Segmenting Anything](https://mlsanity.com/paper/2401.16741/) (2024)

## Abstract (arXiv)

This paper introduces a novel approach to the task of data association within the context of pedestrian tracking, by introducing a two-stage learning scheme to match pairs of detections. First, a Siamese convolutional neural network (CNN) is trained to learn descriptors encoding local spatio-temporal structures between the two input image patches, aggregating pixel values and optical flow information. Second, a set of contextual features derived from the position and size of the compared input patches are combined with the CNN output by means of a gradient boosting classifier to generate the final matching probability. This learning approach is validated by using a linear programming based multi-person tracker showing that even a simple and efficient tracker may outperform much more complex models when fed with our learned matching probabilities. Results on publicly available sequences show that our method meets state-of-the-art standards in multiple people tracking.