# DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction

arXiv: https://arxiv.org/abs/2403.02075 · HTML: https://mlsanity.com/paper/2403.02075/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 43 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification](https://mlsanity.com/paper/2302.11813/) (2023)
- [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](https://mlsanity.com/paper/2304.05170/) (2023)
- [GENIE: Higher-Order Denoising Diffusion Solvers](https://mlsanity.com/paper/2210.05475/) (2022)
- [Explicit Visual Prompting for Low-Level Structure Segmentations](https://mlsanity.com/paper/2303.10883/) (2023)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)

## Built on by (most cited first, 30 of 59)

- [Multiple Object Tracking as ID Prediction](https://mlsanity.com/paper/2403.16848/) (2024)
- [SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation](https://mlsanity.com/paper/2504.04519/) (2025)
- [Omnidirectional Multi-Object Tracking](https://mlsanity.com/paper/2503.04565/) (2025)
- [RAM: Recover Any 3D Human Motion in-the-Wild](https://mlsanity.com/paper/2603.19929/) (2026)
- [DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation](https://mlsanity.com/paper/2503.13957/) (2025)
- [TrackSSM: A General Motion Predictor by State-Space Model](https://mlsanity.com/paper/2409.00487/) (2024)
- [No Train Yet Gain: Towards Generic Multi-Object Tracking in Sports and Beyond](https://mlsanity.com/paper/2506.01373/) (2025)
- [Associate Everything Detected: Facilitating Tracking-by-Detection to the Unknown](https://mlsanity.com/paper/2409.09293/) (2024)
- [GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models](https://mlsanity.com/paper/2408.07259/) (2024)
- [Video Individual Counting for Moving Drones](https://mlsanity.com/paper/2503.10701/) (2025)
- [CAMELTrack: Context-Aware Multi-cue ExpLoitation for Online Multi-Object Tracking](https://mlsanity.com/paper/2505.01257/) (2025)
- [OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback](https://mlsanity.com/paper/2511.00510/) (2025)
- [NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking](https://mlsanity.com/paper/2509.02111/) (2025)
- [Motion-Aware Vision-Reference Alignment for Referring Multi-Object Tracking](https://mlsanity.com/paper/2511.17681/) (2025)
- [History-Aware Transformation of ReID Features for Multiple Object Tracking](https://mlsanity.com/paper/2503.12562/) (2025)
- [SportMamba: Adaptive Non-Linear Multi-Object Tracking with State Space Models for Team Sports](https://mlsanity.com/paper/2506.03335/) (2025)
- [One Graph to Track Them All: Dynamic GNNs for Single- and Multi-View Tracking](https://mlsanity.com/paper/2507.08494/) (2025)
- [DDTracking: A Deep Generative Framework for Diffusion MRI Tractography with Streamline Local-Global Spatiotemporal Modeling](https://mlsanity.com/paper/2508.04568/) (2025)
- [Distractor-Aware Memory-Based Visual Object Tracking](https://mlsanity.com/paper/2509.13864/) (2025)
- [DepTR-MOT: Unveiling the Potential of Depth-Informed Trajectory Refinement for Multi-Object Tracking](https://mlsanity.com/paper/2509.17323/) (2025)
- [AR-MOT: Autoregressive Multi-object Tracking](https://mlsanity.com/paper/2601.01925/) (2026)
- [Hypergraph-State Collaborative Reasoning for Multi-Object Tracking](https://mlsanity.com/paper/2604.12665/) (2026)
- [DINTR: Tracking via Diffusion-based Interpolation](https://mlsanity.com/paper/2410.10053/) (2024)
- [FastTrackTr:Towards Fast Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2411.15811/) (2024)
- [Towards long-term player tracking with graph hierarchies and domain-specific features](https://mlsanity.com/paper/2502.21242/) (2025)
- [Depth-Aware Scoring and Hierarchical Alignment for Multiple Object Tracking](https://mlsanity.com/paper/2506.00774/) (2025)
- [A Novel Tuning Method for Real-time Multiple-Object Tracking Utilizing Thermal Sensor with Complexity Motion Pattern](https://mlsanity.com/paper/2507.02408/) (2025)
- [Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos](https://mlsanity.com/paper/2508.01730/) (2025)
- [GRASPTrack: Geometry-Reasoned Association via Segmentation and Projection for Multi-Object Tracking](https://mlsanity.com/paper/2508.08117/) (2025)
- [EPIPTrack: Rethinking Prompt Modeling with Explicit and Implicit Prompts for Multi-Object Tracking](https://mlsanity.com/paper/2510.13235/) (2025)

## Abstract (arXiv)

In Multiple Object Tracking, objects often exhibit non-linear motion of acceleration and deceleration, with irregular direction changes. Tacking-by-detection (TBD) trackers with Kalman Filter motion prediction work well in pedestrian-dominant scenarios but fall short in complex situations when multiple objects perform non-linear and diverse motion simultaneously. To tackle the complex non-linear motion, we propose a real-time diffusion-based MOT approach named DiffMOT. Specifically, for the motion predictor component, we propose a novel Decoupled Diffusion-based Motion Predictor (D2MP). It models the entire distribution of various motion presented by the data as a whole. It also predicts an individual object's motion conditioning on an individual's historical motion information. Furthermore, it optimizes the diffusion process with much fewer sampling steps. As a MOT tracker, the DiffMOT is real-time at 22.7FPS, and also outperforms the state-of-the-art on DanceTrack and SportsMOT datasets with 62.3% and 76.2% in HOTA metrics, respectively. To the best of our knowledge, DiffMOT is the first to introduce a diffusion probabilistic model into the MOT to tackle non-linear motion prediction.