# OvarNet: Towards Open-vocabulary Object Attribute Recognition

arXiv: https://arxiv.org/abs/2301.09506 · HTML: https://mlsanity.com/paper/2301.09506/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 23 Jan 2023.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [PhraseCut: Language-based Image Segmentation in the Wild](https://mlsanity.com/paper/2008.01187/) (2020)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [Deep Imbalanced Learning for Face Recognition and Attribute Prediction](https://mlsanity.com/paper/1806.00194/) (2018)
- [Disentangling Visual Embeddings for Attributes and Objects](https://mlsanity.com/paper/2205.08536/) (2022)
- [Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts](https://mlsanity.com/paper/1803.06049/) (2018)
- [Label, Verify, Correct: A Simple Few Shot Object Detection Method](https://mlsanity.com/paper/2112.05749/) (2021)
- [Zero Shot Recognition with Unreliable Attributes](https://mlsanity.com/paper/1409.4327/) (2014)
- [Open-vocabulary Attribute Detection](https://mlsanity.com/paper/2211.12914/) (2022)

## Built on by (most cited first, 30 of 43)

- [RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model](https://mlsanity.com/paper/2306.16269/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations](https://mlsanity.com/paper/2403.20312/) (2024)
- [AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation](https://mlsanity.com/paper/2309.00096/) (2023)
- [Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning](https://mlsanity.com/paper/2311.11646/) (2023)
- [Advancing Textual Prompt Learning with Anchored Attributes](https://mlsanity.com/paper/2412.09442/) (2024)
- [Continuous Remote Sensing Image Super-Resolution based on Context Interaction in Implicit Function Space](https://mlsanity.com/paper/2302.08046/) (2023)
- [Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance](https://mlsanity.com/paper/2407.14032/) (2024)
- [ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images](https://mlsanity.com/paper/2410.24001/) (2024)
- [RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models](https://mlsanity.com/paper/2507.06231/) (2025)
- [ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation](https://mlsanity.com/paper/2311.13258/) (2023)
- [Is CLIP the main roadblock for fine-grained open-world perception?](https://mlsanity.com/paper/2404.03539/) (2024)
- [Open World Object Detection: A Survey](https://mlsanity.com/paper/2410.11301/) (2024)
- [RSBuilding: Towards General Remote Sensing Image Building Extraction and Change Detection with Foundation Model](https://mlsanity.com/paper/2403.07564/) (2024)
- [HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging](https://mlsanity.com/paper/2501.16751/) (2025)
- [FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance](https://mlsanity.com/paper/2407.05578/) (2024)
- [EagleVision: Object-level Attribute Multimodal LLM for Remote Sensing](https://mlsanity.com/paper/2503.23330/) (2025)
- [Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation](https://mlsanity.com/paper/2510.00681/) (2025)
- [Transformation-Invariant Network for Few-Shot Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2303.06817/) (2023)
- [PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning](https://mlsanity.com/paper/2511.04601/) (2025)
- [Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptation](https://mlsanity.com/paper/2411.02057/) (2024)
- [DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolution](https://mlsanity.com/paper/2405.16071/) (2024)
- [Detailed Object Description with Controllable Dimensions](https://mlsanity.com/paper/2411.19106/) (2024)
- [Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition](https://mlsanity.com/paper/2412.13947/) (2024)
- [Compositional Caching for Training-free Open-vocabulary Attribute Detection](https://mlsanity.com/paper/2503.19145/) (2025)
- [Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model](https://mlsanity.com/paper/2403.00376/) (2024)
- [MarsSeg: Mars Surface Semantic Segmentation with Multi-level Extractor and Connector](https://mlsanity.com/paper/2404.04155/) (2024)
- [ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modeling](https://mlsanity.com/paper/2408.04102/) (2024)
- [LOWA: Localize Objects in the Wild with Attributes](https://mlsanity.com/paper/2305.20047/) (2023)

## Abstract (arXiv)

In this paper, we consider the problem of simultaneously detecting objects and inferring their visual attributes in an image, even for those with no manual annotations provided at the training stage, resembling an open-vocabulary scenario. To achieve this goal, we make the following contributions: (i) we start with a naive two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes; (ii) we combine all available datasets and train with a federated strategy to finetune the CLIP model, aligning the visual representation with attributes, additionally, we investigate the efficacy of leveraging freely available online image-caption pairs under weakly supervised learning; (iii) in pursuit of efficiency, we train a Faster-RCNN type model end-to-end with knowledge distillation, that performs class-agnostic object proposals and classification on semantic categories and attributes with classifiers generated from a text encoder; Finally, (iv) we conduct extensive experiments on VAW, MS-COCO, LSA, and OVAD datasets, and show that recognition of semantic category and attributes is complementary for visual scene understanding, i.e., jointly training object detection and attributes prediction largely outperform existing approaches that treat the two tasks independently, demonstrating strong generalization ability to novel attributes and categories.