# Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking

arXiv: https://arxiv.org/abs/2103.16178 · HTML: https://mlsanity.com/paper/2103.16178/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 79 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [DeeperGCN: All You Need to Train Deeper GCNs](https://mlsanity.com/paper/2006.07739/) (2020)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [Learning Combinatorial Embedding Networks for Deep Graph Matching](https://mlsanity.com/paper/1904.00597/) (2019)
- [Deep Affinity Network for Multiple Object Tracking](https://mlsanity.com/paper/1810.11780/) (2018)
- [Learning by tracking: Siamese CNN for robust target association](https://mlsanity.com/paper/1604.07866/) (2016)
- [Online Multi-Object Tracking with Dual Matching Attention Networks](https://mlsanity.com/paper/1902.00749/) (2019)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [How To Train Your Deep Multi-Object Tracker](https://mlsanity.com/paper/1906.06618/) (2019)
- [RetinaTrack: Online Single Stage Joint Detection and Tracking](https://mlsanity.com/paper/2003.13870/) (2020)
- [FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking](https://mlsanity.com/paper/1904.04989/) (2019)
- [Lifted Disjoint Paths with Application in Multiple Object Tracking](https://mlsanity.com/paper/2006.14550/) (2020)
- [Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor](https://mlsanity.com/paper/1504.02340/) (2015)
- [Exploit the Connectivity: Multi-Object Tracking with TrackletNet](https://mlsanity.com/paper/1811.07258/) (2018)
- [A Study of Lagrangean Decompositions and Dual Ascent Solvers for Graph Matching](https://mlsanity.com/paper/1612.05476/) (2016)
- [GCNNMatch: Graph Convolutional Neural Networks for Multi-Object Tracking via Sinkhorn Normalization](https://mlsanity.com/paper/2010.00067/) (2020)

## Built on by (most cited first, 30 of 47)

- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detection](https://mlsanity.com/paper/2203.06398/) (2022)
- [Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking](https://mlsanity.com/paper/2308.00783/) (2023)
- [MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking](https://mlsanity.com/paper/2303.10404/) (2023)
- [Unifying Short and Long-Term Tracking with Graph Hierarchies](https://mlsanity.com/paper/2212.03038/) (2022)
- [SparseTrack: Multi-Object Tracking by Performing Scene Decomposition based on Pseudo-Depth](https://mlsanity.com/paper/2306.05238/) (2023)
- [Immortal Tracker: Tracklet Never Dies](https://mlsanity.com/paper/2111.13672/) (2021)
- [DiffusionTrack: Diffusion Model For Multi-Object Tracking](https://mlsanity.com/paper/2308.09905/) (2023)
- [Tracking Objects as Pixel-wise Distributions](https://mlsanity.com/paper/2207.05518/) (2022)
- [Towards Discriminative Representation: Multi-view Trajectory Contrastive Learning for Online Multi-object Tracking](https://mlsanity.com/paper/2203.14208/) (2022)
- [Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2112.09828/) (2021)
- [Densely Constrained Depth Estimator for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10047/) (2022)
- [Towards Generalizable Multi-Object Tracking](https://mlsanity.com/paper/2406.00429/) (2024)
- [Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking](https://mlsanity.com/paper/2105.02426/) (2021)
- [Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation](https://mlsanity.com/paper/2212.01568/) (2022)
- [Online Multi-Object Tracking with Unsupervised Re-Identification Learning and Occlusion Estimation](https://mlsanity.com/paper/2201.01297/) (2022)
- [Learning of Global Objective for Network Flow in Multi-Object Tracking](https://mlsanity.com/paper/2203.16210/) (2022)
- [SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks](https://mlsanity.com/paper/2403.19474/) (2024)
- [Multi-Object Tracking and Segmentation via Neural Message Passing](https://mlsanity.com/paper/2207.07454/) (2022)
- [LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds](https://mlsanity.com/paper/2308.09908/) (2023)
- [DeconfuseTrack:Dealing with Confusion for Multi-Object Tracking](https://mlsanity.com/paper/2403.02767/) (2024)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Multi-Scene Generalized Trajectory Global Graph Solver with Composite Nodes for Multiple Object Tracking](https://mlsanity.com/paper/2312.08951/) (2023)
- [Ensemble Quadratic Assignment Network for Graph Matching](https://mlsanity.com/paper/2403.06457/) (2024)
- [PD-SORT: Occlusion-Robust Multi-Object Tracking Using Pseudo-Depth Cues](https://mlsanity.com/paper/2501.11288/) (2025)
- [CURSOR: Scalable Mixed-Order Hypergraph Matching with CUR Decomposition](https://mlsanity.com/paper/2402.16594/) (2024)
- [NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking](https://mlsanity.com/paper/2509.02111/) (2025)
- [Tracking Objects with 3D Representation from Videos](https://mlsanity.com/paper/2306.05416/) (2023)
- [Permutation-Invariant Graph Partitioning:How Graph Neural Networks Capture Structural Interactions?](https://mlsanity.com/paper/2312.08671/) (2023)

## Abstract (arXiv)

Data association across frames is at the core of Multiple Object Tracking (MOT) task. This problem is usually solved by a traditional graph-based optimization or directly learned via deep learning. Despite their popularity, we find some points worth studying in current paradigm: 1) Existing methods mostly ignore the context information among tracklets and intra-frame detections, which makes the tracker hard to survive in challenging cases like severe occlusion. 2) The end-to-end association methods solely rely on the data fitting power of deep neural networks, while they hardly utilize the advantage of optimization-based assignment methods. 3) The graph-based optimization methods mostly utilize a separate neural network to extract features, which brings the inconsistency between training and inference. Therefore, in this paper we propose a novel learnable graph matching method to address these issues. Briefly speaking, we model the relationships between tracklets and the intra-frame detections as a general undirected graph. Then the association problem turns into a general graph matching between tracklet graph and detection graph. Furthermore, to make the optimization end-to-end differentiable, we relax the original graph matching into continuous quadratic programming and then incorporate the training of it into a deep graph network with the help of the implicit function theorem. Lastly, our method GMTracker, achieves state-of-the-art performance on several standard MOT datasets. Our code will be available at https://github.com/jiaweihe1996/GMTracker .