# An Iterative Feedback Mechanism for Improving Natural Language Class Descriptions in Open-Vocabulary Object Detection

arXiv: https://arxiv.org/abs/2503.17285 · HTML: https://mlsanity.com/paper/2503.17285/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2212.09506/) (2022)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [Part-Stacked CNN for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1512.08086/) (2015)
- [Multi-Modal Classifiers for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.05493/) (2023)
- [Improving Zero-shot Generalization and Robustness of Multi-modal Models](https://mlsanity.com/paper/2212.01758/) (2022)
- [LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors](https://mlsanity.com/paper/2402.04630/) (2024)
- [OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision](https://mlsanity.com/paper/2405.17913/) (2024)
- [Evolving Interpretable Visual Classifiers with Large Language Models](https://mlsanity.com/paper/2404.09941/) (2024)
- [Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation](https://mlsanity.com/paper/2401.06167/) (2024)
- [If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions](https://mlsanity.com/paper/2403.16442/) (2024)
- [Multi-modal Attribute Prompting for Vision-Language Models](https://mlsanity.com/paper/2403.00219/) (2024)
- [IntCoOp: Interpretability-Aware Vision-Language Prompt Tuning](https://mlsanity.com/paper/2406.13683/) (2024)
- [Training-free Boost for Open-Vocabulary Object Detection with Confidence Aggregation](https://mlsanity.com/paper/2404.08603/) (2024)
- [Open-Vocabulary Object Detection with Meta Prompt Representation and Instance Contrastive Optimization](https://mlsanity.com/paper/2403.09433/) (2024)
- [Quantifying and Enabling the Interpretability of CLIP-like Models](https://mlsanity.com/paper/2409.06579/) (2024)
- [An Application-Agnostic Automatic Target Recognition System Using Vision Language Models](https://mlsanity.com/paper/2411.03491/) (2024)

## Abstract (arXiv)

Recent advances in open-vocabulary object detection models will enable Automatic Target Recognition systems to be sustainable and repurposed by non-technical end-users for a variety of applications or missions. New, and potentially nuanced, classes can be defined with natural language text descriptions in the field, immediately before runtime, without needing to retrain the model. We present an approach for improving non-technical users' natural language text descriptions of their desired targets of interest, using a combination of analysis techniques on the text embeddings, and proper combinations of embeddings for contrastive examples. We quantify the improvement that our feedback mechanism provides by demonstrating performance with multiple publicly-available open-vocabulary object detection models.