mlsanityData to 7 Oct 2026

SILC: Improving Vision Language Pretraining with Self-Distillation

arXiv 2310.13355 ↗First posted 20 Oct 2023cs.CVcited by 66

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Image-Text pretraining on web-scale image caption datasets has become the default recipe for open vocabulary classification and retrieval models thanks to the success of CLIP and its variants. Several works have also used CLIP features for dense prediction tasks and have shown the emergence of open-set abilities. However, the contrastive objective used by these models only focuses on image-text alignment and does not incentivise image feature learning for dense prediction tasks. In this work, we introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation, while also providing improvements on image-level tasks such as classification and retrieval. SILC models sets a new state of the art for zero-shot classification, few shot classification, image and text retrieval, zero-shot segmentation, and open vocabulary segmentation. We further show that SILC features greatly benefit open vocabulary detection, captioning and visual question answering.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 72 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

68 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Image Captioners Are Scalable Vision Learners Too13 Jun
  2. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  3. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec

Other references · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  13. 2021Scaling Vision Transformerscited by 993
  14. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  15. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  17. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  18. 2022Language-driven Semantic Segmentationcited by 711
  19. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  21. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  22. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  23. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  24. 2021Extract Free Dense Labels from CLIPcited by 580
  25. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  26. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  28. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  29. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  30. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  31. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  32. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  33. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  34. 2023Data Filtering Networkscited by 247
  35. 2019Zero-Shot Semantic Segmentationcited by 245
  36. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  37. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  38. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 23; 2025: 23; 2026: 19.

66 papers.

Sort:

By area:

  1. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  2. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  3. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  4. 2025Scaling Language-Free Visual Representation Learningcited by 62
  5. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  6. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  7. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  8. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  9. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  10. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  11. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  12. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
38 more
  1. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 20
  2. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 20
  3. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  4. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 18
  5. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
  6. 2024How to Benchmark Vision Foundation Models for Semantic Segmentation?cited by 15
  7. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  8. 2024Spot-Compose: A Framework for Open-Vocabulary Object Retrieval and Drawer Manipulation in Point Cloudscited by 14
  9. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 14
  10. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 12
  11. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  12. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 12
  13. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  14. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 9
  15. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMscited by 7
  16. 2024Multilingual Vision-Language Pre-training for the Remote Sensing Domaincited by 7
  17. 2025Test-Time Visual In-Context Tuningcited by 7
  18. 2024OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentationcited by 6
  19. 2026Efficient Universal Perception Encodercited by 6
  20. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  21. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  22. 2026The Spatial Blindspot of Vision-Language Modelscited by 5
  23. 2025What Holds Back Open-Vocabulary Segmentation?cited by 4
  24. 2024Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Taskscited by 3
  25. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  26. 2025ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Predictioncited by 2
  27. 2025One Patch to Caption Them All: A Unified Zero-Shot Captioning Frameworkcited by 2
  28. 2025Boosting Medical Visual Understanding From Multi-Granular Language Learningcited by 2
  29. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  30. 2026Revisiting Multi-Task Visual Representation Learningcited by 2
  31. 2026Text-Conditional JEPA for Learning Semantically Rich Visual Representationscited by 2
  32. 2026Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compressioncited by 2
  33. 2024Semantic Compositions Enhance Vision-Language Contrastive Learningcited by 1
  34. 2025VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformercited by 1
  35. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  36. 2025Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentationcited by 1
  37. 2025Multilingual Vision-Language Models, A Surveycited by 1
  38. 2025Efficient Discriminative Joint Encoders for Large Scale Vision-Language Rerankingcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
  2. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  3. 2024DreamLIP: Language-Image Pre-training with Long Captions
  4. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition
  5. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
  6. 2023Improved baselines for vision-language pre-training
  7. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
  8. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends