mlsanityData to 7 Oct 2026

ZegOT: Zero-shot Segmentation Through Optimal Transport of Text Prompts

arXiv 2301.12171 ↗First posted 28 Jan 2023cs.CV cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent success of large-scale Contrastive Language-Image Pre-training (CLIP) has led to great promise in zero-shot semantic segmentation by transferring image-text aligned knowledge to pixel-level classification. However, existing methods usually require an additional image encoder or retraining/tuning the CLIP module. Here, we propose a novel Zero-shot segmentation with Optimal Transport (ZegOT) method that matches multiple text prompts with frozen image embeddings through optimal transport. In particular, we introduce a novel Multiple Prompt Optimal Transport Solver (MPOT), which is designed to learn an optimal mapping between multiple text prompts and visual feature maps of the frozen image encoder hidden layers. This unique mapping method facilitates each of the multiple text prompts to effectively focus on distinct visual semantic attributes. Through extensive experiments on benchmark datasets, we show that our method achieves the state-of-the-art (SOTA) performance over existing Zero-shot Semantic Segmentation (ZS3) approaches.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

32 of 45 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
20 more
  1. 2021GPT Understands, Toocited by 715
  2. 2022Language-driven Semantic Segmentationcited by 711
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  5. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  7. 2015Optimal Transport for Domain Adaptationcited by 457
  8. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  9. 2019Zero-Shot Semantic Segmentationcited by 245
  10. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  11. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  12. 2022Unified Vision and Language Prompt Learningcited by 145
  13. 2019Gromov-Wasserstein Learning for Graph Matching and Node Embeddingcited by 125
  14. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  15. 2019Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matchingcited by 102
  16. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  17. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  18. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Modelscited by 64
  19. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  20. 2022Prompt-Matched Semantic Segmentationcited by 22

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 4; 2025: 4.

16 papers.

Sort:

By area:

  1. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  2. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  3. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  4. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 38
  5. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  6. 2023Visual Tuningcited by 34
  7. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  8. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  9. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  10. 2023LLM-driven Multimodal Target Volume Contouring in Radiation Oncologycited by 20
  11. 2023OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognitioncited by 16
  12. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentationcited by 6
4 more
  1. 2024Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Modelcited by 4
  2. 2025MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classificationcited by 3
  3. 2025OV-COAST: Cost Aggregation with Optimal Transport for Open-Vocabulary Semantic Segmentationcited by 2
  4. 2025Learning local and global prototypes with optimal transport for unsupervised anomaly detection and localizationcited by 0