Data to 7 Oct 2026

F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

arXiv 2209.15639 ↗First posted 30 Sep 2022cs.CVcited by 149

Not on the map. Why?

Cited by 40 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models. F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining. Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is a strong region classifier. We finetune only the detector head and combine the detector and VLM outputs for each region at inference time. F-VLM shows compelling scaling behavior and achieves +6.5 mask AP improvement over the previous state of the art on novel categories of LVIS open-vocabulary detection benchmark. In addition, we demonstrate very competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection, in addition to significant training speed-up and compute savings. Code will be released at the https://sites.google.com/view/f-vlm/home

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 50 of its 59 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

50 of 59 references.

References · 50

2015Deep Residual Learning for Image Recognitioncited by 51,809
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
2020End-to-End Object Detection with Transformerscited by 7,384
2017Mask R-CNNcited by 7,259
2016Feature Pyramid Networks for Object Detectioncited by 5,021
2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530

38 more

2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
2021Grounded Language-Image Pre-trainingcited by 1,147
2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
2021Extract Free Dense Labels from CLIPcited by 580
2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
2020Open-Vocabulary Object Detection Using Captionscited by 427
2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
2018Zero-Shot Object Detectioncited by 189
2015Weakly Supervised Deep Detection Networkscited by 183
2021Combined Scaling for Zero-shot Transfer Learningcited by 182
2015Learning Visual Features from Large Weakly Supervised Datacited by 165
2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
2020Learning Visual Representations with Caption Annotationscited by 123
2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
2021Learning Open-World Object Proposals without Learning to Classifycited by 109
2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
2015Webly Supervised Learning of Convolutional Networkscited by 95
2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
2014Zero Shot Recognition with Unreliable Attributescited by 39
2018Zero-Shot Object Detection by Hybrid Region Embeddingcited by 32
2020Synthesizing the Unseen for Zero-shot Object Detectioncited by 32
2019Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detectioncited by 31
2020Background Learnable Cascade for Zero-Shot Object Detectioncited by 31
2022Proper Reuse of Image Classification Features Improves Object Detectioncited by 20
2015DeepBox: Learning Objectness with Convolutional Networkscited by 14
2020What leads to generalization of object proposals?cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 51; 2024: 44; 2025: 32; 2026: 21.

149 papers.

Sort:

By area:

2023Vision-Language Models for Vision Tasks: A Surveycited by 834
2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
2023Language to Rewards for Robotic Skill Synthesiscited by 307
2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
2023Towards Open Vocabulary Learning: A Surveycited by 165
2023Transformer-Based Visual Segmentation: A Surveycited by 124
2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
2023Contextual Object Detection with Multimodal Large Language Modelscited by 100

38 more

2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignmentcited by 71
2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 53
2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignmentcited by 44
2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
2025YOLOE: Real-Time Seeing Anythingcited by 36
2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
2023Tokenize Anything via Promptingcited by 31
2024On the Foundations of Earth and Climate Foundation Modelscited by 31
2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
2023LP-OVOD: Open-Vocabulary Object Detection by Linear Probingcited by 29
2024InstaGen: Enhancing Object Detection by Training on Synthetic Datasetcited by 26
2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformercited by 26
2023Taming Self-Training for Open-Vocabulary Object Detectioncited by 25
2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
2024COCONut: Modernizing COCO Segmentationcited by 24
2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detectioncited by 24
2024Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentationcited by 23
2024OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervisioncited by 22
2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Modelscited by 20
2023Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real Worldcited by 18
2023Three ways to improve feature alignment for open vocabulary detectioncited by 17
2024Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillationcited by 17
2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 17
2023GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learningcited by 15

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way