# Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection

arXiv: https://arxiv.org/abs/2406.00510 · HTML: https://mlsanity.com/paper/2406.00510/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 50 references that are in the map.
- First posted on arXiv 1 Jun 2024.

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [PromptDet: Towards Open-vocabulary Detection using Uncurated Images](https://mlsanity.com/paper/2203.16513/) (2022)
- [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023)
- [Probabilistic two-stage detection](https://mlsanity.com/paper/2103.07461/) (2021)
- [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023)
- [Aligning Pretraining for Detection via Object-Level Contrastive Learning](https://mlsanity.com/paper/2106.02637/) (2021)
- [Exploiting Unlabeled Data with Vision and Language Models for Object Detection](https://mlsanity.com/paper/2207.08954/) (2022)
- [Learning Object-Language Alignments for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2211.14843/) (2022)

## Built on by (most cited first, 24 of 24)

- [Cross-Modal Attention Calibration for LVLM Hallucination Mitigation](https://mlsanity.com/paper/2501.01926/) (2025)
- [MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation](https://mlsanity.com/paper/2602.21397/) (2026)
- [VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.22664/) (2025)
- [NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2603.21069/) (2026)
- [Boosting Open-Vocabulary Object Detection by Handling Background Samples](https://mlsanity.com/paper/2410.08645/) (2024)
- [Prompt Diffusion Robustifies Any-Modality Prompt Learning](https://mlsanity.com/paper/2410.20164/) (2024)
- [MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2510.14792/) (2025)
- [DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2601.20064/) (2026)
- [Prompt-Driven Continual Graph Learning](https://mlsanity.com/paper/2502.06327/) (2025)
- [LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching](https://mlsanity.com/paper/2506.23502/) (2025)
- [VK-Det: Visual Knowledge Guided Prototype Learning for Open-Vocabulary Aerial Object Detection](https://mlsanity.com/paper/2511.18075/) (2025)
- [A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection](https://mlsanity.com/paper/2601.11910/) (2026)
- [ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models](https://mlsanity.com/paper/2603.17079/) (2026)
- [Disentangling Granularity: An Implicit Inductive Bias in Factorized VAEs](https://mlsanity.com/paper/2505.24684/) (2025)
- [Understanding What Is Not Said:Referring Remote Sensing Image Segmentation with Scarce Expressions](https://mlsanity.com/paper/2510.22760/) (2025)
- [Auto-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2512.16077/) (2025)
- [IntRec: Intent-based Retrieval with Contrastive Refinement](https://mlsanity.com/paper/2602.17639/) (2026)
- [GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection](https://mlsanity.com/paper/2603.27014/) (2026)
- [Neutral-Reference Prompting for Vision-Language Models](https://mlsanity.com/paper/2605.15615/) (2026)
- [COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection](https://mlsanity.com/paper/2605.27116/) (2026)
- [LV-OSD: Language-Vision-Complementary Open-Set Object Detection](https://mlsanity.com/paper/2605.28271/) (2026)
- [Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs](https://mlsanity.com/paper/2606.05916/) (2026)
- [ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification](https://mlsanity.com/paper/2606.09360/) (2026)
- [Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases](https://mlsanity.com/paper/2609.09462/) (2026)

## Abstract (arXiv)

Open vocabulary object detection (OVD) aims at seeking an optimal object detector capable of recognizing objects from both base and novel categories. Recent advances leverage knowledge distillation to transfer insightful knowledge from pre-trained large-scale vision-language models to the task of object detection, significantly generalizing the powerful capabilities of the detector to identify more unknown object categories. However, these methods face significant challenges in background interpretation and model overfitting and thus often result in the loss of crucial background knowledge, giving rise to sub-optimal inference performance of the detector. To mitigate these issues, we present a novel OVD framework termed LBP to propose learning background prompts to harness explored implicit background knowledge, thus enhancing the detection performance w.r.t. base and novel categories. Specifically, we devise three modules: Background Category-specific Prompt, Background Object Discovery, and Inference Probability Rectification, to empower the detector to discover, represent, and leverage implicit object knowledge explored from background proposals. Evaluation on two benchmark datasets, OV-COCO and OV-LVIS, demonstrates the superiority of our proposed method over existing state-of-the-art approaches in handling the OVD tasks.