# LaMOT: Language-Guided Multi-Object Tracking arXiv: https://arxiv.org/abs/2406.08324 ยท HTML: https://mlsanity.com/paper/2406.08324/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 38 of its 47 references that are in the map. - First posted on arXiv 12 Jun 2024. ## Builds on (most cited first, 30 of 38) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019) - [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018) - [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019) - [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021) - [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017) - [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016) - [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021) - [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016) - [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020) - [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016) - [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021) - [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020) - [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021) - [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020) - [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020) - [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022) - [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022) - [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021) - [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022) - [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023) - [MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors](https://mlsanity.com/paper/2211.09791/) (2022) - [TAO: A Large-Scale Benchmark for Tracking Any Object](https://mlsanity.com/paper/2005.10356/) (2020) - [Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification](https://mlsanity.com/paper/2302.11813/) (2023) - [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](https://mlsanity.com/paper/2304.05170/) (2023) - [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022) ## Built on by (most cited first, 9 of 9) - [Unhackable Temporal Rewarding for Scalable Video MLLMs](https://mlsanity.com/paper/2502.12081/) (2025) - [RefAV: Towards Planning-Centric Scenario Mining](https://mlsanity.com/paper/2505.20981/) (2025) - [ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking](https://mlsanity.com/paper/2505.20381/) (2025) - [Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2510.10671/) (2025) - [IoT-Brain: Grounding LLMs for Semantic-Spatial Sensor Scheduling](https://mlsanity.com/paper/2604.08033/) (2026) - [ViewSAM: Learning View-aware Cross-modal Semantics for Weakly Supervised Cross-view Referring Multi-Object Tracking](https://mlsanity.com/paper/2605.02638/) (2026) - [STORM: End-to-End Referring Multi-Object Tracking in Videos](https://mlsanity.com/paper/2604.10527/) (2026) - [Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes](https://mlsanity.com/paper/2604.19318/) (2026) - [COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking](https://mlsanity.com/paper/2605.14795/) (2026) ## Abstract (arXiv) Vision-Language MOT is a crucial tracking problem and has drawn increasing attention recently. It aims to track objects based on human language commands, replacing the traditional use of templates or pre-set information from training sets in conventional tracking tasks. Despite various efforts, a key challenge lies in the lack of a clear understanding of why language is used for tracking, which hinders further development in this field. In this paper, we address this challenge by introducing Language-Guided MOT, a unified task framework, along with a corresponding large-scale benchmark, termed LaMOT, which encompasses diverse scenarios and language descriptions. Specially, LaMOT comprises 1,660 sequences from 4 different datasets and aims to unify various Vision-Language MOT tasks while providing a standardized evaluation platform. To ensure high-quality annotations, we manually assign appropriate descriptive texts to each target in every video and conduct careful inspection and correction. To the best of our knowledge, LaMOT is the first benchmark dedicated to Language-Guided MOT. Additionally, we propose a simple yet effective tracker, termed LaMOTer. By establishing a unified task framework, providing challenging benchmarks, and offering insights for future algorithm design and evaluation, we expect to contribute to the advancement of research in Vision-Language MOT. We will release the data at https://github.com/Nathan-Li123/LaMOT.