mlsanityData to 7 Oct 2026

SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation

arXiv 2211.14813 ↗First posted 27 Nov 2022cs.CV cs.AIcited by 162

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, the contrastive language-image pre-training, e.g., CLIP, has demonstrated promising results on various downstream tasks. The pre-trained model can capture enriched visual concepts for images by learning from a large scale of text-image data. However, transferring the learned visual knowledge to open-vocabulary semantic segmentation is still under-explored. In this paper, we propose a CLIP-based model named SegCLIP for the topic of open-vocabulary segmentation in an annotation-free manner. The SegCLIP achieves segmentation based on ViT and the main idea is to gather patches with learnable centers to semantic regions through training on text-image pairs. The gathering operation can dynamically capture the semantic groups, which can be used to generate the final segmentation results. We further propose a reconstruction loss on masked patches and a superpixel-based KL loss with pseudo-labels to enhance the visual representation. Experimental results show that our model achieves comparable or superior segmentation accuracy on the PASCAL VOC 2012 (+0.3% mIoU), PASCAL Context (+2.3% mIoU), and COCO (+2.2% mIoU) compared with baselines. We release the code at https://github.com/ArrowLuo/SegCLIP.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

61 of 73 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2016Pyramid Scene Parsing Networkcited by 2,472
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  17. 2021Grounded Language-Image Pre-trainingcited by 1,147
  18. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  19. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  20. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  21. 2022Language-driven Semantic Segmentationcited by 711
  22. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  23. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  24. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  25. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  26. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  27. 2021Extract Free Dense Labels from CLIPcited by 580
  28. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  29. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  30. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  31. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  32. 2021Image Segmentation Using Text and Image Promptscited by 509
  33. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  34. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  35. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  36. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  37. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  38. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 41; 2024: 60; 2025: 38; 2026: 23.

162 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  4. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  5. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  6. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  7. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  8. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
  9. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  10. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  11. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  12. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
38 more
  1. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  2. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  3. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  4. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  5. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  6. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  7. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  8. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  9. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  10. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  11. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  12. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 43
  13. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  14. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  15. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  16. 2023CLIP in Medical Imaging: A Surveycited by 40
  17. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  18. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  19. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  20. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  21. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  22. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 33
  23. 2024Transfer CLIP for Generalizable Image Denoisingcited by 29
  24. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 28
  25. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  26. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  27. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  28. 2024OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understandingcited by 22
  29. 2023A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Applicationcited by 21
  30. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  31. 2024Image-Text Co-Decomposition for Text-Supervised Semantic Segmentationcited by 21
  32. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  33. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  34. 2023Extending CLIP's Image-Text Alignment to Referring Image Segmentationcited by 19
  35. 2023TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Trainingcited by 19
  36. 2024T-VSL: Text-Guided Visual Sound Source Localization in Mixturescited by 19
  37. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  38. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17

50 most cited of 162 All 162 (JSON)