mlsanityData to 7 Oct 2026

Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation

arXiv 2406.00670 ↗First posted 2 Jun 2024cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-trained vision-language models, e.g., CLIP, have been successfully applied to zero-shot semantic segmentation. Existing CLIP-based approaches primarily utilize visual features from the last layer to align with text embeddings, while they neglect the crucial information in intermediate layers that contain rich object details. However, we find that directly aggregating the multi-level visual features weakens the zero-shot ability for novel classes. The large differences between the visual features from different layers make these features hard to align well with the text embeddings. We resolve this problem by introducing a series of independent decoders to align the multi-level visual features with the text embeddings in a cascaded way, forming a novel but simple framework named Cascade-CLIP. Our Cascade-CLIP is flexible and can be easily applied to existing zero-shot semantic segmentation methods. Experimental results show that our simple Cascade-CLIP achieves superior zero-shot performance on segmentation benchmarks, like COCO-Stuff, Pascal-VOC, and Pascal-Context. Our code is available at: https://github.com/HVision-NKU/Cascade-CLIP

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

51 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentation30 Sep
  2. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation16 Mar
  3. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  8. 2016Pyramid Scene Parsing Networkcited by 2,472
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
35 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  5. 2021Extract Free Dense Labels from CLIPcited by 580
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  7. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  8. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  9. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  10. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  11. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  13. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  14. 2019Zero-Shot Semantic Segmentationcited by 245
  15. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  16. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  17. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  18. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  19. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  20. 2020Strip Pooling: Rethinking Spatial Pooling for Scene Parsingcited by 114
  21. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  22. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  23. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  24. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 94
  25. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  26. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  27. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  28. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  29. 2020AlignSeg: Feature-Aligned Segmentation Networkscited by 53
  30. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  31. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41
  32. 2023Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Networkcited by 40
  33. 2023TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentationcited by 14
  34. 2023Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentationcited by 11
  35. 2023MVP-SEG: Multi-View Prompt Learning for Open-Vocabulary Semantic Segmentationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 6; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  2. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  3. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  4. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  5. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  6. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  7. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  8. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  9. 2025FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Textcited by 7
  10. 2024High-Quality Mask Tuning Matters for Open-Vocabulary Segmentationcited by 5
  11. 2025Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignmentcited by 5
  12. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classificationcited by 5
5 more
  1. 2025SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detectioncited by 3
  2. 2025AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Promptingcited by 2
  3. 2026CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentationcited by 1
  4. 2024DoubleCCA: Improving Foundation Model Group Robustness with Random Sentence Embeddingscited by 0
  5. 2026Fine-grained CLIP fine-tuning with self-annotated region alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
  2. 2023[CLS] Token is All You Need for Zero-Shot Semantic Segmentation
  3. 2023CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation
  4. 2025Split Matching for Inductive Zero-shot Semantic Segmentation
  5. 2024S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models
  6. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
  7. 2024MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment
  8. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer