mlsanityData to 7 Oct 2026

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

arXiv 2310.01403 ↗First posted 2 Oct 2023cs.CVcited by 114

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating vision transformers (ViTs), have exhibited remarkable generalization ability in zero-shot image classification. However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. In this paper, we embark on an in-depth analysis of the region-language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose an approach named CLIPSelf, which adapts the image-level recognition ability of CLIP ViT to local image regions without needing any region-text pairs. CLIPSelf empowers ViTs to distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the corresponding image crop. With the enhanced CLIP ViTs, we achieve new state-of-the-art performance on open-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at https://github.com/wusize/CLIPSelf.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 54 of its 69 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

54 of 69 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2017Mask R-CNNcited by 7,259
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2015Fast R-CNNcited by 2,930
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
38 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  4. 2021Grounded Language-Image Pre-trainingcited by 1,147
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  6. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  7. 2018Panoptic Segmentationcited by 796
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  10. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  11. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  12. 2021Extract Free Dense Labels from CLIPcited by 580
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  14. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  15. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  16. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  17. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  18. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  19. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  20. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  21. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  22. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  23. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  24. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  25. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  26. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  27. 2014Simultaneous Detection and Segmentationcited by 158
  28. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  29. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  30. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  31. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  32. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  33. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  34. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  35. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  36. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  37. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  38. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labelscited by 71

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 36; 2025: 45; 2026: 29.

114 papers.

Sort:

By area:

  1. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  2. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  3. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  4. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  5. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  6. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  7. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  8. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  9. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  10. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  11. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  12. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
38 more
  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  3. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  4. 2024OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understandingcited by 22
  5. 2024OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervisioncited by 22
  6. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 18
  7. 2024CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understandingcited by 16
  8. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  9. 2026Vision Transformers Need More Than Registerscited by 14
  10. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  11. 2024Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Modelscited by 13
  12. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  13. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 12
  14. 2025Locality-Aware Zero-Shot Human-Object Interaction Detectioncited by 12
  15. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 12
  16. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  17. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  18. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  19. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  20. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  21. 2023CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentationcited by 9
  22. 2024O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representationcited by 9
  23. 2024DreamRelation: Bridging Customization and Relation Generationcited by 9
  24. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  25. 2024YOLO-UniOW: Efficient Universal Open-World Object Detectioncited by 9
  26. 2025A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detectioncited by 9
  27. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  28. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  29. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 9
  30. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  31. 2025Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detectioncited by 8
  32. 2025Rethinking Cross-Generator Image Forgery Detection through DINOv3cited by 8
  33. 2023OV-VG: A Benchmark for Open-Vocabulary Visual Groundingcited by 7
  34. 2023Rethinking Evaluation Metrics of Open-Vocabulary Segmentaioncited by 7
  35. 2025Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIPcited by 7
  36. 2025Seeing What Matters: Empowering CLIP with Patch Generation-to-Selectioncited by 7
  37. 2025Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Imagescited by 7
  38. 2024Tuning-free Universally-Supervised Semantic Segmentationcited by 6

50 most cited of 114 All 114 (JSON)