# TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking

arXiv: https://arxiv.org/abs/2101.04206 · HTML: https://mlsanity.com/paper/2101.04206/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 62 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 30 of 33)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [Joint Monocular 3D Vehicle Detection and Tracking](https://mlsanity.com/paper/1811.10742/) (2018)
- [Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Plans](https://mlsanity.com/paper/2001.00735/) (2020)
- [FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking](https://mlsanity.com/paper/1904.04989/) (2019)
- [Robust Multi-Modality Multi-Object Tracking](https://mlsanity.com/paper/1909.03850/) (2019)
- [Track to Reconstruct and Reconstruct to Track](https://mlsanity.com/paper/1910.00130/) (2019)
- [Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor](https://mlsanity.com/paper/1504.02340/) (2015)
- [3D Multi-Object Tracking: A Baseline and New Evaluation Metrics](https://mlsanity.com/paper/1907.03961/) (2019)
- [How would surround vehicles move? A Unified Framework for Maneuver Classification and Motion Prediction](https://mlsanity.com/paper/1801.06523/) (2018)
- [Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking](https://mlsanity.com/paper/1802.09298/) (2018)
- [JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset](https://mlsanity.com/paper/2002.08397/) (2020)
- [Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds](https://mlsanity.com/paper/1904.07537/) (2019)
- [Mono-Camera 3D Multi-Object Tracking Using Deep Learning Detections and PMBM Filtering](https://mlsanity.com/paper/1802.09975/) (2018)
- [Combined Image- and World-Space Tracking in Traffic Scenes](https://mlsanity.com/paper/1809.07357/) (2018)
- [FANTrack: 3D Multi-Object Tracking with Feature Association Network](https://mlsanity.com/paper/1905.02843/) (2019)
- [Accurate Single Stage Detector Using Recurrent Rolling Convolution](https://mlsanity.com/paper/1704.05776/) (2017)
- [DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation](https://mlsanity.com/paper/1909.12471/) (2019)
- [End-to-end Learning of Multi-sensor 3D Tracking by Detection](https://mlsanity.com/paper/1806.11534/) (2018)
- [No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles using Cameras & LiDARs](https://mlsanity.com/paper/1802.08755/) (2018)
- [Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection](https://mlsanity.com/paper/1604.04693/) (2016)
- [PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling](https://mlsanity.com/paper/2003.07847/) (2020)
- [Factor Graph based 3D Multi-Object Tracking in Point Clouds](https://mlsanity.com/paper/2008.05309/) (2020)

## Built on by (most cited first, 21 of 21)

- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation](https://mlsanity.com/paper/2312.08952/) (2023)
- [TransCenter: Transformers with Dense Representations for Multiple-Object Tracking](https://mlsanity.com/paper/2103.15145/) (2021)
- [DiffusionTrack: Diffusion Model For Multi-Object Tracking](https://mlsanity.com/paper/2308.09905/) (2023)
- [Tracking Objects as Pixel-wise Distributions](https://mlsanity.com/paper/2207.05518/) (2022)
- [ADA-Track++: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association](https://mlsanity.com/paper/2405.08909/) (2024)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM](https://mlsanity.com/paper/2210.16204/) (2022)
- [3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking](https://mlsanity.com/paper/2308.06635/) (2023)
- [Neural Enhanced Belief Propagation for Multiobject Tracking](https://mlsanity.com/paper/2212.08340/) (2022)
- [Neural Enhanced Belief Propagation for Data Association in Multiobject Tracking](https://mlsanity.com/paper/2203.09948/) (2022)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Tracking Objects with 3D Representation from Videos](https://mlsanity.com/paper/2306.05416/) (2023)
- [Occlusion-Aware Detection and Re-ID Calibrated Network for Multi-Object Tracking](https://mlsanity.com/paper/2308.15795/) (2023)
- [Hypergraph-State Collaborative Reasoning for Multi-Object Tracking](https://mlsanity.com/paper/2604.12665/) (2026)
- [A Scalable Multi-Robot Framework for Decentralized and Asynchronous Perception-Action-Communication Loops](https://mlsanity.com/paper/2309.10164/) (2023)
- [An Approximate Dynamic Programming Framework for Occlusion-Robust Multi-Object Tracking](https://mlsanity.com/paper/2405.15137/) (2024)
- [Hierarchical IoU Tracking based on Interval](https://mlsanity.com/paper/2406.13271/) (2024)

## Abstract (arXiv)

This study follows many classical approaches to multi-object tracking (MOT) that model the problem using dynamic graphical data structures, and adapts this formulation to make it amenable to modern neural networks. Our main contributions in this work are the creation of a framework based on dynamic undirected graphs that represent the data association problem over multiple timesteps, and a message passing graph neural network (MPNN) that operates on these graphs to produce the desired likelihood for every association therein. We also provide solutions and propositions for the computational problems that need to be addressed to create a memory-efficient, real-time, online algorithm that can reason over multiple timesteps, correct previous mistakes, update beliefs, and handle missed/false detections. To demonstrate the efficacy of our approach, we only use the 2D box location and object category ID to construct the descriptor for each object instance. Despite this, our model performs on par with state-of-the-art approaches that make use of additional sensors, as well as multiple hand-crafted and/or learned features. This illustrates that given the right problem formulation and model design, raw bounding boxes (and their kinematics) from any off-the-shelf detector are sufficient to achieve competitive tracking results on challenging MOT benchmarks.