mlsanityData to 7 Oct 2026

Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning

arXiv 2212.04994 ↗First posted 9 Dec 2022cs.CVcited by 115

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabulary zero-shot segmentation on 4 different segmentation benchmarks: Pascal VOC, Pascal Context, COCO Stuff and ADE20K. Furthermore, we show that PACL is also applicable to image-level predictions and when used with a CLIP backbone, provides a general improvement in zero-shot classification accuracy compared to CLIP, across a suite of 12 image classification datasets.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

49 of 64 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2016Pyramid Scene Parsing Networkcited by 2,472
  11. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
37 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  13. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  14. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2022Language-driven Semantic Segmentationcited by 711
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  18. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  19. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  20. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  21. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  22. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  23. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  24. 2019Zero-Shot Semantic Segmentationcited by 245
  25. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  26. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  27. 2019Connecting Vision and Language with Localized Narrativescited by 225
  28. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  29. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  30. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  31. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  32. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 111
  33. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  34. 2019SegSort: Segmentation by Discriminative Sorting of Segmentscited by 96
  35. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  36. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  37. 2016A Comprehensive Survey on Cross-modal Retrievalcited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 27; 2024: 36; 2025: 32; 2026: 20.

115 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023LERF: Language Embedded Radiance Fieldscited by 538
  3. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  4. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  5. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  6. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  7. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  8. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  9. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  10. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  11. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  12. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
38 more
  1. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  2. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  3. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  4. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  5. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  6. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  7. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  8. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  9. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  10. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  11. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  12. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  13. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  14. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  15. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  16. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  17. 2023What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentationcited by 24
  18. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentationcited by 21
  19. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  20. 2023Mitigating Test-Time Bias for Fair Image Retrievalcited by 20
  21. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  22. 2024Textual Query-Driven Mask Transformer for Domain Generalized Segmentationcited by 20
  23. 2023Top-Down Visual Attention from Analysis by Synthesiscited by 19
  24. 2024T-VSL: Text-Guided Visual Sound Source Localization in Mixturescited by 19
  25. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  26. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17
  27. 2024Open Panoramic Segmentationcited by 17
  28. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  29. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14
  30. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  31. 2023Unsupervised Open-Vocabulary Object Localization in Videoscited by 13
  32. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 12
  33. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  34. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  35. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 11
  36. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  37. 2023CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentationcited by 9
  38. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9

50 most cited of 115 All 115 (JSON)