mlsanityData to 7 Oct 2026

Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation

arXiv 2407.08268 ↗First posted 11 Jul 2024cs.CVcited by 61

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

CLIP, as a vision-language model, has significantly advanced Open-Vocabulary Semantic Segmentation (OVSS) with its zero-shot capabilities. Despite its success, its application to OVSS faces challenges due to its initial image-level alignment training, which affects its performance in tasks requiring detailed local context. Our study delves into the impact of CLIP's [CLS] token on patch feature correlations, revealing a dominance of "global" patches that hinders local feature discrimination. To overcome this, we propose CLIPtrase, a novel training-free semantic segmentation strategy that enhances local feature awareness through recalibrated self-correlation among patches. This approach demonstrates notable improvements in segmentation accuracy and the ability to maintain semantic coherence across objects.Experiments show that we are 22.3% ahead of CLIP on average on 9 segmentation benchmarks, outperforming existing state-of-the-art training-free methods.The code are made publicly available at: https://github.com/leaves162/CLIPtrase.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 72 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

65 of 72 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models7 Jul
  2. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentation30 Sep
  3. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learning29 Aug
  4. 2023[CLS] Token is All You Need for Zero-Shot Semantic Segmentation13 Apr
  5. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training12 Apr
  6. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  7. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  8. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  9. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  10. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Segment Anythingcited by 8,158
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2023Vision Transformers Need Registerscited by 844
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  12. 2022Language-driven Semantic Segmentationcited by 711
38 more
  1. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  2. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  5. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  7. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  8. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  9. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  10. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  11. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  12. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  13. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  14. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  15. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  16. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  17. 2021Semi-supervised Semantic Segmentation with Directional Context-aware Consistencycited by 112
  18. 2021ResLT: Residual Learning for Long-tailed Recognitioncited by 96
  19. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  20. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  21. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 94
  22. 2021Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentationcited by 87
  23. 2023Hierarchical Dense Correlation Distillation for Few-Shot Segmentationcited by 84
  24. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  25. 2020Generalized Few-shot Semantic Segmentationcited by 59
  26. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  27. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  28. 2023Open-Vocabulary Segmentation with Semantic-Assisted Calibrationcited by 49
  29. 2022Generalized Parametric Contrastive Learningcited by 45
  30. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  31. 2023Leveraging Hidden Positives for Unsupervised Semantic Segmentationcited by 40
  32. 2023Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Networkcited by 40
  33. 2021PFENet++: Boosting Few-shot Semantic Segmentation with the Noise-filtered Context-aware Prior Maskcited by 31
  34. 2024EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentationcited by 27
  35. 2023Learning Context-aware Classifier for Semantic Segmentationcited by 23
  36. 2022DecoupleNet: Decoupled Network for Domain Adaptive Semantic Segmentationcited by 22
  37. 2023Source-Free Domain Adaptive Fundus Image Segmentation with Class-Balanced Mean Teachercited by 21
  38. 2023SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentationcited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 23; 2026: 28.

61 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  2. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  3. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  4. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  5. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 33
  6. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  7. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 28
  8. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  9. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  10. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 23
  11. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  12. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 14
38 more
  1. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 14
  2. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  3. 2025Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidancecited by 9
  4. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  5. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  6. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 9
  7. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  8. 2025TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Modelscited by 7
  9. 2025Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Imagescited by 7
  10. 2025Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transportcited by 6
  11. 2025Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspectivecited by 5
  12. 2026Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLMcited by 4
  13. 2025VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labelscited by 3
  14. 2025CalibCLIP: Contextual Calibration of Dominant Semantics for Text-Driven Image Retrievalcited by 3
  15. 2026Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentationcited by 3
  16. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  17. 2025Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentationcited by 2
  18. 2025ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Predictioncited by 2
  19. 2026Enabling Training-Free Text-Based Remote Sensing Segmentationcited by 2
  20. 2026Open-Vocabulary Domain Generalization in Urban-Scene Segmentationcited by 2
  21. 2026Rethinking MLLM Itself as a Segmenter with a Single Segmentation Tokencited by 2
  22. 2026Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentationcited by 2
  23. 2024MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Image Segmentationcited by 1
  24. 2024LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentationcited by 1
  25. 2025Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentationcited by 1
  26. 2025Collaborative Learning with Multiple Foundation Models for Source-Free Domain Adaptationcited by 1
  27. 2026Locality-Attending Vision Transformercited by 1
  28. 2026Multimodal Dataset Distillation via Phased Teacher Modelscited by 1
  29. 2026DINO Soars: DINOv3 for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagerycited by 1
  30. 2026VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inferencecited by 1
  31. 2026Mechanisms of Object Localization in Vision-Language Modelscited by 1
  32. 2025Bridge Feature Matching and Cross-Modal Alignment with Mutual-filtering for Zero-shot Anomaly Detectioncited by 0
  33. 2025Interpreting ResNet-based CLIP via Neuron-Attention Decompositioncited by 0
  34. 2025Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentationcited by 0
  35. 2025ARM: A Learnable, Plug-and-Play Module for CLIP-based Open-vocabulary Semantic Segmentationcited by 0
  36. 2026Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagerycited by 0
  37. 2026Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Compositioncited by 0
  38. 2026MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentationcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
  2. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
  3. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
  4. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
  5. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation
  6. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation
  7. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
  8. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models