mlsanityData to 13 Oct 2026

Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation

arXiv 2403.08426 ↗First posted 13 Mar 2024cs.CV cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The pre-trained vision-language model, exemplified by CLIP, advances zero-shot semantic segmentation by aligning visual features with class embeddings through a transformer decoder to generate semantic masks. Despite its effectiveness, prevailing methods within this paradigm encounter challenges, including overfitting on seen classes and small fragmentation in masks. To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information.Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings. Moreover, to circumvent noisy alignments from the vision part due to its redundant nature, we introduce route attention into self-attention for finding visual consensus, thereby enhancing semantic consistency within the same object. Equipped with a vision-language prompting strategy, our approach significantly boosts the generalization capacity of segmentation models for unseen classes. Experimental results underscore the effectiveness of our approach, showcasing mIoU gains of 4.5 on the PASCAL VOC 2012 and 3.6 on the COCO-Stuff 164k for unseen classes compared with the state-of-the-art methods.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 78 of its 98 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

78 of 98 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model10 Mar
  2. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentation30 Sep
  3. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learning29 Aug
  4. 2023Delving into Shape-aware Zero-shot Semantic Segmentation17 Apr
  5. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation16 Mar
  6. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models4 Nov
  7. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  8. 2022Prompt Distribution Learning6 May
  9. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  10. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  11. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
1 more
  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 65

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  7. 2023Segment Anythingcited by 8,195
  8. 2023Visual Instruction Tuningcited by 7,827
  9. 2020End-to-End Object Detection with Transformerscited by 7,396
  10. 2016Layer Normalizationcited by 5,624
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,344
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,204
  4. 2021Vision Transformers for Dense Predictioncited by 1,795
  5. 2022Visual Prompt Tuningcited by 1,677
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,164
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,150
  9. 2022Planning-oriented Autonomous Drivingcited by 1,074
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 861
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 726
  13. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 718
  14. 2022Language-driven Semantic Segmentationcited by 715
  15. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  16. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 627
  17. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 615
  18. 2021Extract Free Dense Labels from CLIPcited by 582
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  20. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  21. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  22. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 470
  23. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  24. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 304
  25. 2023Demystifying CLIP Datacited by 267
  26. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  27. 2019Zero-Shot Semantic Segmentationcited by 245
  28. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  29. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  30. 2021Embodied Intelligence via Learning and Evolutioncited by 155
  31. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  32. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  33. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 106
  34. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  35. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  36. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 94
  37. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  38. 2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  3. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosiscited by 9
  4. 2025Few-Shot Adaptation of Training-Free Foundation Model for 3D Medical Image Segmentationcited by 5
  5. 2025Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Modelscited by 2
  6. 2025Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation
  2. 2025Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation
  3. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
  4. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
  5. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation
  6. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
  7. 2023[CLS] Token is All You Need for Zero-Shot Semantic Segmentation
  8. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation