mlsanityData to 7 Oct 2026

ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference

arXiv 2407.12442 ↗First posted 17 Jul 2024cs.CVcited by 112

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Despite the success of large-scale pretrained Vision-Language Models (VLMs) especially CLIP in various open-vocabulary tasks, their application to semantic segmentation remains challenging, producing noisy segmentation maps with mis-segmented regions. In this paper, we carefully re-investigate the architecture of CLIP, and identify residual connections as the primary source of noise that degrades segmentation quality. With a comparative analysis of statistical properties in the residual connection and the attention output across different pretrained models, we discover that CLIP's image-text contrastive training paradigm emphasizes global features at the expense of local discriminability, leading to noisy segmentation results. In response, we propose ClearCLIP, a novel approach that decomposes CLIP's representations to enhance open-vocabulary semantic segmentation. We introduce three simple modifications to the final layer: removing the residual connection, implementing the self-self attention, and discarding the feed-forward network. ClearCLIP consistently generates clearer and more accurate segmentation maps and outperforms existing approaches across multiple benchmarks, affirming the significance of our discoveries.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 56 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

49 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation2 Jun
  2. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentation30 Sep
  3. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation24 Sep
  4. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation1 May
  5. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training12 Apr
  6. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  7. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
30 more
  1. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  2. 2023Vision Transformers Need Registerscited by 844
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  6. 2021Extract Free Dense Labels from CLIPcited by 580
  7. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  8. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  9. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  12. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  13. 2023Demystifying CLIP Datacited by 265
  14. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  15. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  16. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  17. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  18. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  19. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  20. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  21. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  22. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  23. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  24. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  25. 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
  26. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  27. 2023ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistencycited by 50
  28. 2023Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Networkcited by 40
  29. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  30. 2023SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 43; 2026: 54.

112 papers.

Sort:

By area:

  1. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  2. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  3. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  4. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  5. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  6. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  7. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 33
  8. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  9. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  10. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  11. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 28
  12. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
38 more
  1. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 23
  2. 2025DynamicEarth: How Far are We from Open-Vocabulary Change Detection?cited by 21
  3. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  4. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  5. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  6. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 14
  7. 2025Vision Transformers with Self-Distilled Registerscited by 12
  8. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  9. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  10. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  11. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  12. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 9
  13. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  14. 2024ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentationcited by 8
  15. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  16. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  17. 2025TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Modelscited by 7
  18. 2025Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Imagescited by 7
  19. 2025UniVCD: A New Method for Unsupervised Change Detection in the Open-Vocabulary Eracited by 7
  20. 2025Single Domain Generalization for Few-Shot Counting via Universal Representation Matchingcited by 6
  21. 2025Selective Contrastive Learning for Weakly Supervised Affordance Groundingcited by 6
  22. 2026PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulationcited by 6
  23. 2024High-Quality Mask Tuning Matters for Open-Vocabulary Segmentationcited by 5
  24. 2025VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semanticscited by 5
  25. 2025Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspectivecited by 5
  26. 2025SPECIAL: Zero-shot Hyperspectral Image Classification With CLIPcited by 4
  27. 2025What Holds Back Open-Vocabulary Segmentation?cited by 4
  28. 2025PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learningcited by 4
  29. 2025Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detectioncited by 3
  30. 2026Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentationcited by 3
  31. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  32. 2025Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentationcited by 2
  33. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIPcited by 2
  34. 2025ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Predictioncited by 2
  35. 2025NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentationcited by 2
  36. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  37. 2026DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentationcited by 2
  38. 2026Enabling Training-Free Text-Based Remote Sensing Segmentationcited by 2

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
  2. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
  3. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation
  4. 2026HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
  5. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  6. 2024DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks
  7. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
  8. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels