# A novel Framework for Open-Vocabulary Multi-Object Recognition using CLIP

arXiv: https://arxiv.org/abs/2603.05962 · HTML: https://mlsanity.com/paper/2603.05962/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 8 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 7 of 7)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Open-Vocabulary Universal Image Segmentation with MaskCLIP](https://mlsanity.com/paper/2208.08984/) (2022)
- [MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection](https://mlsanity.com/paper/2407.21465/) (2024)
- [A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2503.10152/) (2025)
- [Exploring PCA-based feature representations of image pixels via CNN to enhance food image segmentation](https://mlsanity.com/paper/2411.01469/) (2024)

## Abstract (arXiv)

To address the limitations of existing open-vocabulary object recognition methods, including high system complexity, substantial training costs, and limited generalization capability, this paper proposes a novel Open-Vocabulary Object Recognition (OVOR) framework based on a streamlined two-stage strategy: object segmentation followed by recognition. The proposed framework eliminates the need for complex retraining procedures and labor-intensive annotation. After extracting object regions, object-level image embeddings and category-level text embeddings are generated using CLIP, enabling recognition over arbitrary vocabularies. To reduce dependence on CLIP and enhance encoding flexibility, we further introduce a CNN/MLP-based approach that extracts convolutional neural network (CNN) feature maps and employs a multilayer perceptron (MLP) to align visual features with text embeddings. The resulting embeddings are then concatenated for cross-modal matching. Finally, object recognition is performed through similarity matching between image and text embeddings. Experiments on COCO, Pascal VOC, and ADE20K demonstrate that CLIP-based image encoding achieves the highest average AP, outperforming current state-of-the-art methods. Meanwhile, the results reveal the potential of CLIP-independent image encoding as a promising alternative for OVOR.