mlsanityData to 14 Oct 2026

Unsupervised Open-Vocabulary Object Localization in Videos

arXiv 2309.09858 ↗First posted 18 Sep 2023cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization. We propose a method that first localizes objects in videos via an object-centric approach with slot attention and then assigns text to the obtained slots. The latter is achieved by an unsupervised way to read localized semantic information from the pre-trained CLIP model. The resulting video object localization is entirely unsupervised apart from the implicit annotation contained in CLIP, and it is effectively the first unsupervised approach that yields good results on regular video benchmarks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 34 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

30 of 34 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  9. 2020Object-Centric Learning with Slot Attentioncited by 876
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  12. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
18 more
  1. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  2. 2019Video Instance Segmentationcited by 393
  3. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  4. 2021Conditional Object-Centric Learning from Videocited by 248
  5. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  6. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  7. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 166
  8. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  9. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  10. 2020Memory Enhanced Global-Local Aggregation for Video Object Detectioncited by 128
  11. 2021How to represent part-whole hierarchies in a neural networkcited by 123
  12. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  13. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  14. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  15. 2020Toward unsupervised, multi-object discovery in large-scale image collectionscited by 65
  16. 2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
  17. 2022Discovering Objects that Can Movecited by 44
  18. 2021End-to-End Video Object Detection with Spatial-Temporal Transformerscited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  2. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  3. 2024CountGD: Multi-Modal Open-World Countingcited by 59
  4. 2024Adaptive Slot Attention: Object Discovery with Dynamic Slot Numbercited by 34
  5. 2025Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspectivecited by 20
  6. 2025Impossible Videoscited by 16
  7. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  8. 2025CTRL-O: Language-Controllable Object-Centric Visual Representation Learningcited by 12
  9. 2023Appearance-Based Refinement for Object-Centric Motion Segmentationcited by 11
  10. 2024Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approachcited by 5
  11. 2026DiCo: Disentangled Concept Representation for Text-to-image Person Re-identificationcited by 2
  12. 2025Multi-View Slot Attention Using Paraphrased Texts for Face Anti-Spoofingcited by 1
1 more
  1. 2026Mitigating Simplicity Bias in OOD Detection through Object Co-occurrence Analysiscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Video OWL-ViT: Temporally-consistent open-world localization in video
  2. 2024Teaching VLMs to Localize Specific Objects from In-context Examples
  3. 2023Learning Object State Changes in Videos: An Open-World Perspective
  4. 2025Harnessing Object Grounding for Time-Sensitive Video Understanding
  5. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
  6. 2024Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization
  7. 2024Open-Vocabulary Action Localization with Iterative Visual Prompting
  8. 2024Zero-shot Action Localization via the Confidence of Large Vision-Language Models