mlsanityData to 14 Oct 2026

Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models

arXiv 2505.09139 ↗First posted 14 May 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) offer flexible object detection through natural language prompts but suffer from performance variability depending on prompt phrasing. In this paper, we introduce a method for automated prompt refinement using a novel metric called the Contrastive Class Alignment Score (CCAS), which ranks prompts based on their semantic alignment with a target object class while penalizing similarity to confounding classes. Our method generates diverse prompt candidates via a large language model and filters them through CCAS, computed using prompt embeddings from a sentence transformer. We evaluate our approach on challenging object categories, demonstrating that our automatic selection of high-precision prompts improves object detection accuracy without the need for additional model training or labeled data. This scalable and model-agnostic pipeline offers a principled alternative to manual prompt engineering for VLM-based detection systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 32 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

24 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024IPO: Interpretable Prompt Optimization for Vision-Language Models20 Oct
  2. 2022Prompt Distribution Learning6 May
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 21

  1. 2023Segment Anythingcited by 8,234
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  5. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  6. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  7. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  8. 2023Fine-Grained Visual Promptingcited by 82
  9. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  10. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 32
  11. 2024Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Setscited by 25
  12. 2024Language-Driven Active Learning for Diverse Open-Set 3D Object Detectioncited by 15
9 more
  1. 2023Zero-shot Nuclei Detection via Visual-Language Pre-trained Modelscited by 10
  2. 2025Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehiclescited by 9
  3. 2025Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safetycited by 9
  4. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Modelscited by 8
  5. 2024Evaluating Vision-Language Models for Zero-Shot Detection, Classification, and Association of Motorcycles, Passengers, and Helmetscited by 4
  6. 2024Driver Activity Classification Using Generalizable Representations from Vision-Language Modelscited by 3
  7. 2024AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Modelscited by 3
  8. 2024Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safetycited by 2
  9. 2025CamoSAM2: SAM2-oriented Prompt Auto-Refinement for Video Camouflaged Object Detectioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  2. 2026C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes
  3. 2025Learning Generalizable Prompt for CLIP with Class Similarity Knowledge
  4. 2026Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection
  5. 2024PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination
  6. 2026What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
  7. 2025Modeling Variants of Prompts for Vision-Language Models
  8. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?