mlsanityData to 7 Oct 2026

Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels

arXiv 2409.19846 ↗First posted 30 Sep 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which additionally require understanding where the objects are located. In this work, we propose a novel method, PixelCLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods in open-vocabulary semantic segmentation. Project page is available at https://cvlab-kaist.github.io/PixelCLIP

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 67 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

60 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  4. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Segment Anythingcited by 8,158
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  13. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  14. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  15. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  16. 2021Robust fine-tuning of zero-shot modelscited by 785
  17. 2021Extract Free Dense Labels from CLIPcited by 580
  18. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  19. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  20. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  21. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  22. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  23. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  24. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  25. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  26. 2019Zero-Shot Semantic Segmentationcited by 245
  27. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  28. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  29. 2021ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentationcited by 184
  30. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  31. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
  32. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  33. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  34. 2022Rethinking Semantic Segmentation: A Prototype Viewcited by 140
  35. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  36. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  37. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  38. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformerscited by 24
  2. 2025Emergent Outlier View Rejection in Visual Geometry Grounded Transformerscited by 19
  3. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 18
  4. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 18
  5. 2025No time to train! Training-Free Reference-Based Instance Segmentationcited by 9
  6. 2025SAM3-I: Segment Anything with Instructionscited by 9
  7. 2025S^4M: Boosting Semi-Supervised Instance Segmentation with SAMcited by 5
  8. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  9. 2026ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervisioncited by 4
  10. 2026PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentationcited by 4
  11. 2025NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentationcited by 2
  12. 2026No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videoscited by 2
7 more
  1. 2026Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstructioncited by 2
  2. 2025Privacy-Preserving Semantic Segmentation from Ultra-Low-Resolution RGB Inputscited by 1
  3. 2025Exploring Conditions for Diffusion models in Robotic Controlcited by 0
  4. 2026WOW-Seg: A Word-free Open World Segmentation Modelcited by 0
  5. 2026PROSE: Training-Free Egocentric Scene Registration with Vision-Language Modelscited by 0
  6. 2026RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representationcited by 0
  7. 2026GeoFlow: Geo-Aware Modeling of Inter-Area Relationships in Origin-Destination Flow Prediction and Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Open-Vocabulary to Vocabulary-Free Semantic Segmentation
  2. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation
  3. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
  4. 2024Effective SAM Combination for Open-Vocabulary Semantic Segmentation
  5. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements
  6. 2024LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation
  7. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation
  8. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer