mlsanityData to 7 Oct 2026

CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement

arXiv 2310.14108 ↗First posted 21 Oct 2023cs.LG cs.AI cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive language image pretraining (CLIP) is a standard method for training vision-language models. While CLIP is scalable, promptable, and robust to distribution shifts on image classification tasks, it lacks object localization capabilities. This paper studies the following question: Can we augment CLIP training with task-specific vision models from model zoos to improve its visual representations? Towards this end, we leverage open-source task-specific vision models to generate pseudo-labels for an uncurated and noisy image-text dataset. Subsequently, we train CLIP models on these pseudo-labels in addition to the contrastive training on image and text pairs. This simple setup shows substantial improvements of up to 16.3% across different vision tasks, including segmentation, detection, depth estimation, and surface normal estimation. Importantly, these enhancements are achieved without compromising CLIP's existing capabilities, including its proficiency in promptable zero-shot classification.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 60 references that are in the map. First posted on arXiv 21 Oct 2023.

Builds on

47 of 60 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2023Segment Anythingcited by 8,158
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2017Mask R-CNNcited by 7,259
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
35 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2016Pyramid Scene Parsing Networkcited by 2,472
  8. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  9. 2021Vision Transformers for Dense Predictioncited by 1,789
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  13. 2021Grounded Language-Image Pre-trainingcited by 1,147
  14. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  15. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  18. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  19. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  20. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  21. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  22. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  23. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  24. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  25. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  26. 2022A Unified Sequence Interface for Vision Taskscited by 152
  27. 2022SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptationcited by 146
  28. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  29. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  30. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  31. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  32. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  33. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  34. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  35. 2022RangeAugment: Efficient Online Augmentation with Range Learningcited by 8

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 12
  2. 2025FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representationscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning
  2. 2024Contrastive Localized Language-Image Pre-Training
  3. 2023Improved baselines for vision-language pre-training
  4. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters
  6. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
  7. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  8. 2025Data or Language Supervision: What Makes CLIP Better than DINO?