mlsanityData to 7 Oct 2026

Type-to-Track: Retrieve Any Object via Prompt-based Tracking

arXiv 2305.13495 ↗First posted 22 May 2023cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions. We present a new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework (MENDER) using the third-order tensor decomposition. The experiments in five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7% accuracy and 4× speed faster.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 60 of its 80 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

60 of 80 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  10. 2016Modeling Context in Referring Expressionscited by 1,054
  11. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  12. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
38 more
  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  2. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  3. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  4. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  5. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  6. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  7. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  8. 2020Tracking Objects as Pointscited by 483
  9. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  10. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  11. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  12. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  13. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  14. 2019Video Instance Segmentationcited by 391
  15. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  16. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  17. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  18. 2020MOT20: A benchmark for multi object tracking in crowded scenescited by 355
  19. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  20. 2022BoT-SORT: Robust Associations Multi-Pedestrian Trackingcited by 313
  21. 2020Quasi-Dense Similarity Learning for Multiple Object Trackingcited by 220
  22. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  23. 2021Track to Detect and Segment: An Online Multi-Object Trackercited by 156
  24. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  25. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  26. 2017Need for Speed: A Benchmark for Higher Frame Rate Object Trackingcited by 137
  27. 2021Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmarkcited by 124
  28. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  29. 2020Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Trackingcited by 115
  30. 2015A Novel Performance Evaluation Methodology for Single-Target Trackerscited by 113
  31. 2017YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Videocited by 109
  32. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  33. 2023Referring Multi-Object Trackingcited by 91
  34. 2018Long-term Tracking in the Wild: A Benchmarkcited by 55
  35. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  36. 2021Opening up Open-World Trackingcited by 50
  37. 2021Towards General Purpose Vision Systemscited by 46
  38. 2022Tracking Every Thing in the Wildcited by 44

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 10; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023Language Prompt for Autonomous Drivingcited by 110
  2. 2023HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understandingcited by 21
  3. 2024Bootstrapping Referring Multi-Object Trackingcited by 21
  4. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 18
  5. 2025VideoMolmo: Spatio-Temporal Grounding Meets Pointingcited by 14
  6. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  7. 2025OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformercited by 11
  8. 2024LaMOT: Language-Guided Multi-Object Trackingcited by 9
  9. 2024MLS-Track: Multilevel Semantic Interaction in RMOTcited by 7
  10. 2025Cognitive Disentanglement for Referring Multi-Object Trackingcited by 6
  11. 2023REACT: Recognize Every Action Everywhere All At Oncecited by 5
  12. 2025Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Modelscited by 5
12 more
  1. 2023HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videoscited by 4
  2. 2025MEX: Memory-efficient Approach to Referring Multi-Object Trackingcited by 3
  3. 2025SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Groundingcited by 3
  4. 2024Multi-Granularity Language-Guided Training for Multi-Object Trackingcited by 2
  5. 2024FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysiscited by 2
  6. 2024Public Health Advocacy Dataset: A Dataset of Tobacco Usage Videos from Social Mediacited by 2
  7. 2024DINTR: Tracking via Diffusion-based Interpolationcited by 1
  8. 2025GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regionscited by 1
  9. 2025See No Evil: Adversarial Attacks Against Linguistic-Visual Association in Referring Multi-Object Tracking Systemscited by 1
  10. 2025EPIPTrack: Rethinking Prompt Modeling with Explicit and Implicit Prompts for Multi-Object Trackingcited by 1
  11. 2025Multi-animal tracking in Transition: Comparative Insights into Established and Emerging Methodscited by 0
  12. 2026Training-Free Semantic Multi-Object Tracking with Vision-Language Modelscited by 0