mlsanityData to 14 Oct 2026

Decoupling Zero-Shot Semantic Segmentation

arXiv 2112.07910 ↗First posted 15 Dec 2021cs.CVcited by 199

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a classagnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter task performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 22 points on the PASCAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https://github.com/dingjiansw101/ZegFormer.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 65 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

38 of 65 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2017Mask R-CNNcited by 7,277
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  9. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  11. 2016Pyramid Scene Parsing Networkcited by 2,479
  12. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
26 more
  1. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  2. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  3. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  4. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  5. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  6. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  7. 2016FastText.zip: Compressing text classification modelscited by 335
  8. 2019Zero-Shot Semantic Segmentationcited by 245
  9. 2021K-Net: Towards Unified Image Segmentationcited by 212
  10. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  11. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  12. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  13. 2015Learning to Segment Object Candidatescited by 79
  14. 2017Open Vocabulary Scene Parsingcited by 76
  15. 2021Open-World Entity Segmentationcited by 74
  16. 2017Learning to Segment Every Thingcited by 65
  17. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  18. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  19. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41
  20. 2021Zero-Shot Instance Segmentationcited by 40
  21. 2014Zero Shot Recognition with Unreliable Attributescited by 39
  22. 2015Zero-Shot Learning via Joint Latent Similarity Embeddingcited by 39
  23. 2017Recent Advances in Zero-shot Recognitioncited by 28
  24. 2016Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learningcited by 20
  25. 2020From Pixel to Patch: Synthesize Context-aware Features for Zero-shot Semantic Segmentationcited by 11
  26. 2021Conterfactual Generative Zero-Shot Semantic Segmentationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 69; 2024: 63; 2025: 32; 2026: 21.

199 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  3. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  4. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  5. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  7. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  8. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  9. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  10. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  11. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  12. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
38 more
  1. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 179
  2. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  4. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  5. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  6. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  7. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  8. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  9. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  10. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 123
  11. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  12. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  13. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  14. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  15. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  16. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  17. 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
  18. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  19. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  20. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  21. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  22. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  23. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  24. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  25. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  26. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  27. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  28. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentationcited by 57
  29. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  30. 2022A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Predictioncited by 46
  31. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  32. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  33. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentationcited by 40
  34. 2023Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Networkcited by 40
  35. 2023Open-vocabulary Panoptic Segmentation with Embedding Modulationcited by 39
  36. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  37. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 39
  38. 2022Hierarchical Local-Global Transformer for Temporal Sentence Groundingcited by 38

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment
  2. 2024S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models
  3. 2024Label Propagation for Zero-shot Classification with Vision-Language Models
  4. 20253D-PointZshotS: Geometry-Aware 3D Point Cloud Zero-Shot Semantic Segmentation Narrowing the Visual-Semantic Gap
  5. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
  6. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
  7. 2025Data-Efficient Generalization for Zero-shot Composed Image Retrieval
  8. 2023Delving into Shape-aware Zero-shot Semantic Segmentation