mlsanityData to 7 Oct 2026

Referring Multi-Object Tracking

arXiv 2303.03366 ↗First posted 6 Mar 2023cs.CVcited by 91

Not on the map. Why?

Abstract (arXiv)

Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance and outperforms other counterparts. The dataset and code will be available at https://github.com/wudongming97/RMOT.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 54 of its 66 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

54 of 66 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
38 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  2. 2016Simple Online and Realtime Trackingcited by 862
  3. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  6. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  7. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
  8. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  9. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  10. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  11. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  12. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  13. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  14. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  15. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  16. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  17. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  18. 2022MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectorscited by 148
  19. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  20. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  21. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  22. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  23. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  24. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  25. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  26. 2018Actor and Action Video Segmentation from a Sentencecited by 105
  27. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  28. 2020Rethinking the competition between detection and ReID in Multi-Object Trackingcited by 101
  29. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  30. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  31. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  32. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  33. 2017Comprehension-guided referring expressionscited by 52
  34. 2022Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentationcited by 50
  35. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  36. 2018Object Referring in Videos with Language and Human Gazecited by 40
  37. 2021Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentationcited by 32
  38. 2020Video Object Grounding using Semantic Roles in Language Descriptioncited by 25

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 25; 2025: 37; 2026: 11.

91 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  3. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  4. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  5. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  7. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  8. 2023Language Prompt for Autonomous Drivingcited by 110
  9. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  10. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  11. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  12. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
38 more
  1. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  2. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  3. 2024Toward Visual Grounding: A Surveycited by 52
  4. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  5. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  6. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  7. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  8. 2023iKUN: Speak to Trackers without Retrainingcited by 31
  9. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  10. 2023Type-to-Track: Retrieve Any Object via Prompt-based Trackingcited by 24
  11. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  12. 2025Multimodal Referring Segmentation: A Surveycited by 22
  13. 2024Bootstrapping Referring Multi-Object Trackingcited by 21
  14. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusioncited by 20
  15. 2023Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyondcited by 19
  16. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 18
  17. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 17
  18. 2024MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Drivingcited by 17
  19. 2024Referring Atomic Video Action Recognitioncited by 16
  20. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 16
  21. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  22. 2024WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radarcited by 13
  23. 2025RefAV: Towards Planning-Centric Scenario Miningcited by 13
  24. 2023DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusionscited by 12
  25. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  26. 2023ASY-VRNet: Waterway Panoptic Driving Perception Model based on Asymmetric Fair Fusion of Vision and 4D mmWave Radarcited by 11
  27. 2025OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformercited by 11
  28. 2023The 1st-place Solution for CVPR 2023 OpenLane Topology in Autonomous Driving Challengecited by 10
  29. 2025ReferGPT: Towards Zero-Shot Referring Multi-Object Trackingcited by 10
  30. 2024LaMOT: Language-Guided Multi-Object Trackingcited by 9
  31. 2024Multiview Scene Graphcited by 9
  32. 2024ActionVOS: Actions as Prompts for Video Object Segmentationcited by 8
  33. 2025Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semanticscited by 8
  34. 20253EED: Ground Everything Everywhere in 3Dcited by 8
  35. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  36. 2024MLS-Track: Multilevel Semantic Interaction in RMOTcited by 7
  37. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  38. 2024Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentationcited by 6

50 most cited of 91 All 91 (JSON)