mlsanityData to 14 Oct 2026

Retrieval-Augmented Open-Vocabulary Object Detection

arXiv 2404.05687 ↗First posted 8 Apr 2024cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Open-vocabulary object detection (OVD) has been studied with Vision-Language Models (VLMs) to detect novel objects beyond the pre-trained categories. Previous approaches improve the generalization ability to expand the knowledge of the detector, using 'positive' pseudo-labels with additional 'class' names, e.g., sock, iPod, and alligator. To extend the previous methods in two aspects, we propose Retrieval-Augmented Losses and visual Features (RALF). Our method retrieves related 'negative' classes and augments loss functions. Also, visual features are augmented with 'verbalized concepts' of classes, e.g., worn on the feet, handheld music player, and sharp teeth. Specifically, RALF consists of two modules: Retrieval Augmented Losses (RAL) and Retrieval-Augmented visual Features (RAF). RAL constitutes two losses reflecting the semantic similarity with negative vocabularies. In addition, RAF augments visual features with the verbalized concepts from a large language model (LLM). Our experiments demonstrate the effectiveness of RALF on COCO and LVIS benchmark datasets. We achieve improvement up to 3.4 box AP50N on novel categories of the COCO dataset and 3.6 mask APr gains on the LVIS dataset. Code is available at https://github.com/mlvlab/RALF .

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 38 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

36 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Open-Vocabulary Object Detection using Pseudo Caption Labels23 Mar
  2. 2023Aligning Bag of Regions for Open-Vocabulary Object Detection27 Feb
  3. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model28 Mar

Other references · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2017Mask R-CNNcited by 7,277
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2015Fast R-CNNcited by 2,933
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
21 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  2. 2022Language-driven Semantic Segmentationcited by 717
  3. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  4. 2021Extract Free Dense Labels from CLIPcited by 583
  5. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  6. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  7. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  8. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
  9. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  10. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  11. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  12. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 96
  13. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  14. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labelscited by 71
  15. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  16. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  17. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  18. 2020RetrieveGAN: Image Synthesis via Differentiable Patch Retrievalcited by 35
  19. 2023Retrieval-augmented Image Captioningcited by 29
  20. 2022Open Vocabulary Object Detection with Proposal Mining and Prediction Equalizationcited by 23
  21. 2020Texture Memory-Augmented Deep Patch-Based Image Inpaintingcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 7.

20 papers.

Sort:

By area:

  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  3. 2025Anomize: Better Open Vocabulary Video Anomaly Detectioncited by 13
  4. 2025A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detectioncited by 9
  5. 2024Retrieval Augmented Generation for Dynamic Graph Modelingcited by 6
  6. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  7. 2025MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detectioncited by 2
  8. 2026RAID: Retrieval-Augmented Anomaly Detectioncited by 2
  9. 2025TrackingMiM: Efficient Mamba-in-Mamba Serialization for Real-time UAV Object Trackingcited by 1
  10. 2025ODOV: Benchmark the Open-Domain Open-Vocabulary Object Detectioncited by 1
  11. 2024Online Learning via Memory: Retrieval-Augmented Detector Adaptationcited by 0
  12. 2024YOLO-RD: Introducing Relevant and Compact Explicit Knowledge to YOLO by Retriever-Dictionarycited by 0
8 more
  1. 2025ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generationcited by 0
  2. 2025Auto-Vocabulary 3D Object Detectioncited by 0
  3. 2026GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detectioncited by 0
  4. 2026Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Methodcited by 0
  5. 2026Towards Adaptive Open-Set Object Detection via Category-Level Collaboration Knowledge Miningcited by 0
  6. 2026Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detectioncited by 0
  7. 2026VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detectioncited by 0
  8. 2026Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
  2. 2024MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection
  3. 2023What Makes Good Open-Vocabulary Detector: A Disassembling Perspective
  4. 2023How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection
  5. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
  6. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding
  7. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future
  8. 2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection