mlsanityData to 14 Oct 2026

SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference

arXiv 2312.01597 ↗First posted 4 Dec 2023cs.CVcited by 170

Not on the map. Why?

Cited by 50 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in contrastive language-image pretraining (CLIP) have demonstrated strong capabilities in zero-shot classification by aligning visual representations with target text embeddings in an image level. However, in dense prediction tasks, CLIP often struggles to localize visual features within an image and fails to give accurate pixel-level predictions, which prevents it from functioning as a generalized visual foundation model. In this work, we aim to enhance CLIP's potential for semantic segmentation with minimal modifications to its pretrained models. By rethinking self-attention, we surprisingly find that CLIP can adapt to dense prediction tasks by simply introducing a novel Correlative Self-Attention (CSA) mechanism. Specifically, we replace the traditional self-attention block of CLIP vision encoder's last layer by our CSA module and reuse its pretrained projection matrices of query, key, and value, leading to a training-free adaptation approach for CLIP's zero-shot semantic segmentation. Extensive experiments show the advantage of CSA: we obtain a 38.2% average zero-shot mIoU across eight semantic segmentation benchmarks highlighted in this paper, significantly outperforming the existing SoTA's 33.9% and the vanilla CLIP's 14.1%.

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). Builds on 69 of its 81 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

69 of 81 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  12. 2023Segment Anythingcited by 8,234
38 more
  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  14. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  15. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  18. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  19. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  20. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  21. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  22. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  23. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  24. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  25. 2021Extract Free Dense Labels from CLIPcited by 583
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  27. 2023Segment Everything Everywhere All at Oncecited by 548
  28. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  29. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  30. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  31. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  32. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  33. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  34. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  35. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  36. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  37. 2022Vision Transformer with Deformable Attentioncited by 295
  38. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 39; 2025: 63; 2026: 68.

170 papers.

Sort:

By area:

  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  2. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  3. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  4. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  5. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  6. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  7. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  8. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48
  9. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 44
  10. 2024Mamba-R: Vision Mamba ALSO Needs Registerscited by 41
  11. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 39
  12. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 39
38 more
  1. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  2. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  3. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
  4. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  5. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  6. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 32
  7. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 29
  8. 2025Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And Morecited by 28
  9. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  10. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 24
  11. 2025DynamicEarth: How Far are We from Open-Vocabulary Change Detection?cited by 21
  12. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21
  13. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  14. 2025JAFAR: Jack up Any Feature at Any Resolutioncited by 21
  15. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 20
  16. 2024Gramformer: Learning Crowd Counting via Graph-Modulated Transformercited by 19
  17. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  18. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  19. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 16
  20. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  21. 2026Vision Transformers Need More Than Registerscited by 15
  22. 2024Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformerscited by 13
  23. 2025Training-free Geometric Image Editing on Diffusion Modelscited by 13
  24. 2025Vision Transformers with Self-Distilled Registerscited by 12
  25. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  26. 20253DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scalecited by 12
  27. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  28. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentationcited by 10
  29. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 10
  30. 2024Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiencycited by 9
  31. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  32. 2025Register and [CLS] tokens yield a decoupling of local and global features in large ViTscited by 9
  33. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  34. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  35. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  36. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  37. 2026ViT-5: Vision Transformers for The Mid-2020scited by 9
  38. 2024ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentationcited by 8

50 most cited of 170 All 170 (JSON)