VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inference
Not on the map. Why?
Abstract (arXiv)
Pursuing training-free open-vocabulary semantic segmentation in an efficient and generalizable manner remains challenging due to the deep-seated spatial bias in CLIP. To overcome the limitations of existing solutions, this work moves beyond the CLIP-based paradigm and harnesses the recent spatially-aware dino.txt framework to facilitate more efficient and high-quality dense prediction. While dino.txt exhibits robust spatial awareness, we find that the semantic ambiguity of text queries gives rise to severe mismatch within its dense cross-modal interactions. To address this, we introduce Visual-guided Prompt evolution (VIP) to rectify the semantic expressiveness of text queries in dino.txt, unleashing its potential for fine-grained object perception. Towards this end, VIP integrates alias expansion with a visual-guided distillation mechanism to mine valuable semantic cues, which are robustly aggregated in a saliency-aware manner to yield a high-fidelity prediction. Extensive evaluations demonstrate that VIP: 1. surpasses the top-leading methods by 1.4%-8.4% average mIoU, 2. generalizes well to diverse challenging domains, and 3. requires marginal inference time and memory overhead.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 60 of its 70 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
60 of 70 references.
References · 60
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023Segment Anythingcited by 8,234
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2025DINOv3cited by 1,414
- 2020Energy-based Out-of-distribution Detectioncited by 1,299
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
- 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
- 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
- 2021Extract Free Dense Labels from CLIPcited by 583
- 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
- 2022Visual Classification via Description from Large Language Modelscited by 336
- 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
- 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 197
- 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
- 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
- 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
- 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
- 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
- 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
- 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
- 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
- 2018Learning random-walk label propagation for weakly-supervised semantic segmentationcited by 101
- 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 101
- 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
- 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 96
- 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
- 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
- 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
- 2023VDD: Varied Drone Dataset for Semantic Segmentationcited by 40
- 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
- 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 39
- 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
- 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
- 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
- 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 29
- 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 29
- 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 24
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.