FOR: Finetuning for Object Level Open Vocabulary Image Retrieval
Not on the map. Why?
Abstract (arXiv)
As working with large datasets becomes standard, the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain, balancing accuracy and efficiency through additional post-processing. In this work, we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval, which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task, and its coupling within a multi-objective training framework. Together, these design choices result in a significant increase in accuracy, showcasing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally, we demonstrate that FOR is also effective in a semi-supervised setting, achieving impressive results even when only a small portion of the dataset is labeled.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 56 of its 74 references that are in the map. First posted on arXiv 25 Dec 2024.
Builds on
56 of 74 references.
References · 56
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
38 more
- 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
- 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
- 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
- 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
- 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
- 2021Florence: A New Foundation Model for Computer Visioncited by 804
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
- 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
- 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
- 2021Extract Free Dense Labels from CLIPcited by 583
- 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
- 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
- 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
- 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
- 2018Stacked Cross Attention for Image-Text Matchingcited by 416
- 2020Meta Pseudo Labelscited by 374
- 2020Rethinking Pre-training and Self-trainingcited by 352
- 2023Scaling Open-Vocabulary Object Detectioncited by 343
- 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
- 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
- 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
- 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
- 2018Zero-Shot Object Detectioncited by 189
- 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
- 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
- 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
- 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
- 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
- 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
- 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
- 2016Instance-aware Image and Sentence Matching with Selective Multimodal LSTMcited by 43
- 2022X-DETR: A Versatile Architecture for Instance-wise Vision-Language Taskscited by 43
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.