mlsanityData to 14 Oct 2026

CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation

arXiv 2411.10086 ↗First posted 15 Nov 2024cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

Open-vocabulary semantic segmentation aims to assign semantic labels to each pixel without being constrained by a predefined set of categories. While Contrastive Language-Image Pre-training (CLIP) excels in zero-shot classification, it struggles to align image patches with category embeddings because of its incoherent patch correlations. This study reveals that inter-class correlations are the main reason for impairing CLIP's segmentation performance. Accordingly, we propose CorrCLIP, which reconstructs the scope and value of patch correlations. Specifically, CorrCLIP leverages the Segment Anything Model (SAM) to define the scope of patch interactions, reducing inter-class correlations. To mitigate the problem that SAM-generated masks may contain patches belonging to different classes, CorrCLIP incorporates self-supervised models to compute coherent similarity values, suppressing the weight of inter-class correlations. Additionally, we introduce two additional branches to strengthen patch features' spatial details and semantic representation. Finally, we update segmentation maps with SAM-generated masks to improve spatial consistency. Based on the improvement across patch correlations, feature representations, and segmentation maps, CorrCLIP achieves superior performance across eight benchmarks. Codes are available at: https://github.com/zdk258/CorrCLIP.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

62 of 81 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Segment Anythingcited by 8,234
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  5. 2023Vision Transformers Need Registerscited by 871
  6. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  9. 2021Extract Free Dense Labels from CLIPcited by 583
  10. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  11. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  12. 2023Segment Anything in High Qualitycited by 376
  13. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  14. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  15. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  16. 2023Fast Segment Anythingcited by 290
  17. 2023Data Filtering Networkscited by 250
  18. 2019Zero-Shot Semantic Segmentationcited by 245
  19. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  20. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  21. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  22. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  23. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  24. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  25. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  26. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  27. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  28. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  29. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 97
  30. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  31. 2020Semantic Segmentation of Underwater Imagery: Dataset and Benchmarkcited by 95
  32. 2022High-Quality Entity Segmentationcited by 93
  33. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  34. 2017Open Vocabulary Scene Parsingcited by 76
  35. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  36. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  37. 2023Open-Vocabulary Segmentation with Semantic-Assisted Calibrationcited by 49
  38. 2021A Large-Scale Benchmark for Food Image Segmentationcited by 45

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 28.

34 papers.

Sort:

By area:

  1. 2025DynamicEarth: How Far are We from Open-Vocabulary Change Detection?cited by 21
  2. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 16
  3. 2026Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentationcited by 4
  4. 2026Open-Vocabulary Domain Generalization in Urban-Scene Segmentationcited by 4
  5. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  6. 2026ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentationcited by 3
  7. 2026Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentationcited by 2
  8. 2026VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inferencecited by 2
  9. 2026Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentationcited by 2
  10. 2025Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentationcited by 1
  11. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  12. 2026CoCo-SAM3: Harnessing Concept Conflict in Open-Vocabulary Semantic Segmentationcited by 1
22 more
  1. 2026SegRAG: Training-Free Retrieval-Augmented Semantic Segmentationcited by 1
  2. 2025A Survey on Training-free Open-Vocabulary Semantic Segmentationcited by 0
  3. 2025ARM: A Learnable, Plug-and-Play Module for CLIP-based Open-vocabulary Semantic Segmentationcited by 0
  4. 2026Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagerycited by 0
  5. 2026LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentationcited by 0
  6. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?cited by 0
  7. 2026Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentationcited by 0
  8. 2026SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentationcited by 0
  9. 2026ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentationcited by 0
  10. 2026OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentationcited by 0
  11. 2026DouC: Dual-Branch CLIP for Training-Free Open-Vocabulary Segmentationcited by 0
  12. 2026Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentationcited by 0
  13. 2026[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregationcited by 0
  14. 2026Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Modelscited by 0
  15. 2026LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentationcited by 0
  16. 2026ActiveSAM: Fast and Accurate Open-Vocabulary Semantic Segmentation with Frozen SAM 3cited by 0
  17. 2026Rapid-Deployment Crack Measurement Based on SAM3 Semantic-Edge Response Decodingcited by 0
  18. 2026Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentationcited by 0
  19. 2026SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentationcited by 0
  20. 2026Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generatorcited by 0
  21. 2026From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentationcited by 0
  22. 2026A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagescited by 0