# CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation

arXiv: https://arxiv.org/abs/2411.10086 · HTML: https://mlsanity.com/paper/2411.10086/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 81 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Builds on (most cited first, 30 of 62)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)

## Built on by (most cited first, 30 of 34)

- [DynamicEarth: How Far are We from Open-Vocabulary Change Detection?](https://mlsanity.com/paper/2501.12931/) (2025)
- [SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images](https://mlsanity.com/paper/2512.08730/) (2025)
- [Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2602.06333/) (2026)
- [Open-Vocabulary Domain Generalization in Urban-Scene Segmentation](https://mlsanity.com/paper/2602.18853/) (2026)
- [PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2603.21528/) (2026)
- [ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation](https://mlsanity.com/paper/2603.29271/) (2026)
- [Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation](https://mlsanity.com/paper/2603.21386/) (2026)
- [VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inference](https://mlsanity.com/paper/2605.12325/) (2026)
- [Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2606.21863/) (2026)
- [Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2510.23894/) (2025)
- [PowerCLIP: Powerset Alignment for Contrastive Pre-Training](https://mlsanity.com/paper/2511.23170/) (2025)
- [CoCo-SAM3: Harnessing Concept Conflict in Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2604.19648/) (2026)
- [SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation](https://mlsanity.com/paper/2605.17630/) (2026)
- [A Survey on Training-free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2505.22209/) (2025)
- [ARM: A Learnable, Plug-and-Play Module for CLIP-based Open-vocabulary Semantic Segmentation](https://mlsanity.com/paper/2512.24224/) (2025)
- [Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagery](https://mlsanity.com/paper/2601.21159/) (2026)
- [LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2602.05578/) (2026)
- [Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?](https://mlsanity.com/paper/2602.23339/) (2026)
- [Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentation](https://mlsanity.com/paper/2604.02010/) (2026)
- [SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation](https://mlsanity.com/paper/2604.02252/) (2026)
- [ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation](https://mlsanity.com/paper/2604.07021/) (2026)
- [OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2604.08110/) (2026)
- [DouC: Dual-Branch CLIP for Training-Free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2604.24997/) (2026)
- [Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2604.26221/) (2026)
- [[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation](https://mlsanity.com/paper/2605.25821/) (2026)
- [Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models](https://mlsanity.com/paper/2606.04385/) (2026)
- [LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation](https://mlsanity.com/paper/2606.11837/) (2026)
- [ActiveSAM: Fast and Accurate Open-Vocabulary Semantic Segmentation with Frozen SAM 3](https://mlsanity.com/paper/2606.16996/) (2026)
- [Rapid-Deployment Crack Measurement Based on SAM3 Semantic-Edge Response Decoding](https://mlsanity.com/paper/2607.12292/) (2026)
- [Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2608.03991/) (2026)

## Abstract (arXiv)

Open-vocabulary semantic segmentation aims to assign semantic labels to each pixel without being constrained by a predefined set of categories. While Contrastive Language-Image Pre-training (CLIP) excels in zero-shot classification, it struggles to align image patches with category embeddings because of its incoherent patch correlations. This study reveals that inter-class correlations are the main reason for impairing CLIP's segmentation performance. Accordingly, we propose CorrCLIP, which reconstructs the scope and value of patch correlations. Specifically, CorrCLIP leverages the Segment Anything Model (SAM) to define the scope of patch interactions, reducing inter-class correlations. To mitigate the problem that SAM-generated masks may contain patches belonging to different classes, CorrCLIP incorporates self-supervised models to compute coherent similarity values, suppressing the weight of inter-class correlations. Additionally, we introduce two additional branches to strengthen patch features' spatial details and semantic representation. Finally, we update segmentation maps with SAM-generated masks to improve spatial consistency. Based on the improvement across patch correlations, feature representations, and segmentation maps, CorrCLIP achieves superior performance across eight benchmarks. Codes are available at: https://github.com/zdk258/CorrCLIP.