# NetTrack: Tracking Highly Dynamic Objects with a Net

arXiv: https://arxiv.org/abs/2403.11186 · HTML: https://mlsanity.com/paper/2403.11186/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 75 references that are in the map.
- First posted on arXiv 17 Mar 2024.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)

## Built on by (most cited first, 22 of 22)

- [Multiple Object Tracking as ID Prediction](https://mlsanity.com/paper/2403.16848/) (2024)
- [Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking](https://mlsanity.com/paper/2410.01806/) (2024)
- [WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark](https://mlsanity.com/paper/2405.19818/) (2024)
- [Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator](https://mlsanity.com/paper/2408.06927/) (2024)
- [SEAL: Semantic Attention Learning for Long Video Representation](https://mlsanity.com/paper/2412.01798/) (2024)
- [SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking](https://mlsanity.com/paper/2409.11235/) (2024)
- [Prompt-Driven Temporal Domain Adaptation for Nighttime UAV Tracking](https://mlsanity.com/paper/2409.18533/) (2024)
- [Associate Everything Detected: Facilitating Tracking-by-Detection to the Unknown](https://mlsanity.com/paper/2409.09293/) (2024)
- [Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2510.10671/) (2025)
- [AR-MOT: Autoregressive Multi-object Tracking](https://mlsanity.com/paper/2601.01925/) (2026)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking](https://mlsanity.com/paper/2512.02392/) (2025)
- [TrAction: Action Recognition with Sparse Trajectories](https://mlsanity.com/paper/2606.03490/) (2026)
- [EffOWT: Transfer Visual Language Models to Open-World Tracking Efficiently and Effectively](https://mlsanity.com/paper/2504.05141/) (2025)
- [Optimizing Multi-Modality Trackers via Significance-Regularized Tuning](https://mlsanity.com/paper/2508.17488/) (2025)
- [Lattice Boltzmann Model for Learning Real-World Pixel Dynamicity](https://mlsanity.com/paper/2509.16527/) (2025)
- [Zero-Shot Multi-Animal Tracking in the Wild](https://mlsanity.com/paper/2511.02591/) (2025)
- [SelfMOTR: Revisiting MOTR with Self-Generating Detection Priors](https://mlsanity.com/paper/2511.20279/) (2025)
- [BlinkBud: Detecting Hazards from Behind via Sampled Monocular 3D Detection on a Single Earbud](https://mlsanity.com/paper/2512.01366/) (2025)
- [SAMOFT: Robust Multi-Object Tracking via Region and Flow](https://mlsanity.com/paper/2605.09417/) (2026)
- [HieDG: A Hierarchical Discrete Geometry-Guided Framework for Multi-Animal Tracking](https://mlsanity.com/paper/2607.00494/) (2026)
- [Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking](https://mlsanity.com/paper/2609.00924/) (2026)

## Abstract (arXiv)

The complex dynamicity of open-world objects presents non-negligible challenges for multi-object tracking (MOT), often manifested as severe deformations, fast motion, and occlusions. Most methods that solely depend on coarse-grained object cues, such as boxes and the overall appearance of the object, are susceptible to degradation due to distorted internal relationships of dynamic objects. To address this problem, this work proposes NetTrack, an efficient, generic, and affordable tracking framework to introduce fine-grained learning that is robust to dynamicity. Specifically, NetTrack constructs a dynamicity-aware association with a fine-grained Net, leveraging point-level visual cues. Correspondingly, a fine-grained sampler and matching method have been incorporated. Furthermore, NetTrack learns object-text correspondence for fine-grained localization. To evaluate MOT in extremely dynamic open-world scenarios, a bird flock tracking (BFT) dataset is constructed, which exhibits high dynamicity with diverse species and open-world scenarios. Comprehensive evaluation on BFT validates the effectiveness of fine-grained learning on object dynamicity, and thorough transfer experiments on challenging open-world benchmarks, i.e., TAO, TAO-OW, AnimalTrack, and GMOT-40, validate the strong generalization ability of NetTrack even without finetuning. Project page: https://george-zhuang.github.io/nettrack/.