mlsanityData to 7 Oct 2026

Opening up Open-World Tracking

arXiv 2104.11221 ↗First posted 22 Apr 2021cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Tracking and detecting any object, including ones never-seen-before during model training, is a crucial but elusive capability of autonomous systems. An autonomous agent that is blind to never-seen-before objects poses a safety hazard when operating in the real world - and yet this is how almost all current systems work. One of the main obstacles towards advancing tracking any object is that this task is notoriously difficult to evaluate. A benchmark that would allow us to perform an apples-to-apples comparison of existing efforts is a crucial first step towards advancing this important research field. This paper addresses this evaluation deficit and lays out the landscape and evaluation methodology for detecting and tracking both known and unknown objects in the open-world setting. We propose a new benchmark, TAO-OW: Tracking Any Object in an Open World, analyze existing efforts in multi-object tracking, and construct a baseline for this task while highlighting future challenges. We hope to open a new front in multi-object tracking research that will hopefully bring us a step closer to intelligent systems that can operate safely in the real world. https://openworldtracking.github.io/

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 78 of its 127 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

78 of 127 references.

References · 78

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2017Mask R-CNNcited by 7,259
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  11. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  12. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
38 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  2. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  3. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  4. 2016Simple Online and Realtime Trackingcited by 862
  5. 2018Panoptic Segmentationcited by 796
  6. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  7. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  8. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  9. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  10. 2015Towards Open Set Deep Networkscited by 533
  11. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
  12. 2019Video Instance Segmentationcited by 391
  13. 2019Tracking without bells and whistlescited by 382
  14. 2021Towards Open World Object Detectioncited by 297
  15. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  16. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  17. 2014Towards Open World Recognitioncited by 226
  18. 2019MOTS: Multi-Object Tracking and Segmentationcited by 222
  19. 2019Learning a Neural Solver for Multiple Object Trackingcited by 198
  20. 2018Zero-Shot Object Detectioncited by 189
  21. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  22. 2018Reducing Network Agnostophobiacited by 165
  23. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  24. 2020Video Panoptic Segmentationcited by 141
  25. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  26. 2020MOTChallenge: A Benchmark for Single-Camera Multiple Target Trackingcited by 134
  27. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  28. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  29. 2020STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videoscited by 121
  30. 2015A Novel Performance Evaluation Methodology for Single-Target Trackerscited by 113
  31. 2017Dropout Sampling for Robust Object Detection in Open-Set Conditionscited by 112
  32. 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
  33. 2015UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Trackingcited by 98
  34. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 95
  35. 2019RVOS: End-to-End Recurrent Network for Video Object Segmentationcited by 88
  36. 2020A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learningcited by 86
  37. 2018PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentationcited by 81
  38. 2015Learning to Segment Object Candidatescited by 79

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 14; 2024: 18; 2025: 6; 2026: 2.

50 papers.

Sort:

By area:

  1. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  3. 2022BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Videocited by 66
  4. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  5. 2022PROB: Probabilistic Objectness for Open World Object Detectioncited by 57
  6. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  7. 2024Matching Anything by Segmenting Anythingcited by 49
  8. 2021Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlookcited by 45
  9. 2023Towards Open-Vocabulary Video Instance Segmentationcited by 45
  10. 2022Tracking Every Thing in the Wildcited by 44
  11. 2022Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinitycited by 41
  12. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
38 more
  1. 2023OpenVIS: Open-vocabulary Video Instance Segmentationcited by 25
  2. 2023Type-to-Track: Retrieve Any Object via Prompt-based Trackingcited by 24
  3. 2024NetTrack: Tracking Highly Dynamic Objects with a Netcited by 22
  4. 2024Self-Supervised Multi-Object Tracking with Path Consistencycited by 16
  5. 2023UGainS: Uncertainty Guided Anomaly Instance Segmentationcited by 15
  6. 2024Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentationcited by 15
  7. 2023Video OWL-ViT: Temporally-consistent open-world localization in videocited by 13
  8. 2024CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentationcited by 13
  9. 2022Beyond SOT: Tracking Multiple Generic Objects at Oncecited by 11
  10. 2023Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Modelscited by 11
  11. 2022GOOD: Exploring Geometric Cues for Detecting Objects in an Open Worldcited by 10
  12. 2024HASSOD: Hierarchical Adaptive Self-Supervised Object Detectioncited by 10
  13. 2024JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environmentscited by 10
  14. 2023Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenescited by 9
  15. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 9
  16. 2022Robust Online Video Instance Segmentation with Track Queriescited by 8
  17. 2023Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervisioncited by 8
  18. 2024SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Trackingcited by 7
  19. 2024Lidar Panoptic Segmentation in an Open Worldcited by 7
  20. 2022Bridging Images and Videos: A Simple Learning Framework for Large Vocabulary Video Object Detectioncited by 6
  21. 2024SOHES: Self-supervised Open-world Hierarchical Entity Segmentationcited by 6
  22. 2025Zero-Shot 4D Lidar Panoptic Segmentationcited by 6
  23. 2023Video Instance Segmentation in an Open-Worldcited by 5
  24. 2025MoSAM: Motion-Guided Segment Anything Model with Spatial-Temporal Memory Selectioncited by 5
  25. 2024OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioningcited by 4
  26. 2024OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Trackingcited by 4
  27. 2024Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentationcited by 3
  28. 2025Realistic Evaluation of TabPFN v2 in Open Environmentscited by 3
  29. 2024UADet: A Remarkably Simple Yet Effective Uncertainty-Aware Open-Set Object Detection Frameworkcited by 2
  30. 2024Open-World Panoptic Segmentationcited by 2
  31. 2023A survey on deep learning approaches for data integration in autonomous driving systemcited by 1
  32. 2024OCTrack: Benchmarking the Open-Corpus Multi-Object Trackingcited by 1
  33. 2025NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environmentscited by 1
  34. 2024The Progression of Transformers from Language to Vision to MOT: A Literature Review on Multi-Object Tracking with Transformerscited by 0
  35. 2024OW-Rep: Open World Object Detection with Instance Representation Learningcited by 0
  36. 2025EffOWT: Transfer Visual Language Models to Open-World Tracking Efficiently and Effectivelycited by 0
  37. 2026COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigmcited by 0
  38. 2026Towards Unconstrained Human-Object Interactioncited by 0