mlsanityData to 14 Oct 2026

S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions

arXiv 2305.14095 ↗First posted 23 May 2023cs.CV cs.LGcited by 30

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 100 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

73 of 100 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger30 Mar
  2. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Models3 Oct
  3. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec

Other references · 70

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2020Supervised Contrastive Learningcited by 2,829
38 more
  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  3. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  10. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  14. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  15. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  16. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  17. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  18. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  19. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  20. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  21. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  23. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  24. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  25. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  26. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  27. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 197
  28. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  29. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  30. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 134
  31. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  32. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  33. 2020Progressive Identification of True Labels for Partial-Label Learningcited by 102
  34. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  35. 2021Weakly Supervised Contrastive Learningcited by 81
  36. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  37. 2021SciCap: Generating Captions for Scientific Figurescited by 77
  38. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 11; 2026: 9.

30 papers.

Sort:

By area:

  1. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  2. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 19
  3. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  4. 2025GOAL: Global-local Object Alignment Learningcited by 17
  5. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
  6. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 15
  7. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 9
  8. 2024MEDBind: Unifying Language and Multimodal Medical Data Embeddingscited by 8
  9. 2024Robust Semi-Supervised Learning in Open Environmentscited by 8
  10. 2024S-MolSearch: 3D Semi-supervised Contrastive Learning for Bioactive Molecule Searchcited by 5
  11. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  12. 2025Learning Shared Representations from Unpaired Datacited by 4
18 more
  1. 2025Open Ad-hoc Categorization with Contextualized Feature Learningcited by 3
  2. 2026SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transportcited by 3
  3. 2024Pseudo-labelling meets Label Smoothing for Noisy Partial Label Learningcited by 1
  4. 2024Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotationscited by 1
  5. 2024Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniquescited by 1
  6. 2025ProgRoCC: A Progressive Approach to Rough Crowd Countingcited by 1
  7. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimizationcited by 1
  8. 2026Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIPcited by 1
  9. 2026Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learningcited by 1
  10. 2024Transmission Line Defect Detection Based on UAV Patrol Images and Vision-language Pretrainingcited by 0
  11. 2025SemiOccam: A Robust Semi-Supervised Image Recognition Network Using Sparse Labelscited by 0
  12. 2025VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensingcited by 0
  13. 2026Group Contrastive Learning for Weakly Paired Multimodal Datacited by 0
  14. 2026AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifierscited by 0
  15. 2026Robust Remote Sensing Image-Text Retrieval with Noisy Correspondencecited by 0
  16. 2026Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Modelscited by 0
  17. 2026Earth Science Foundation Models: From Perception to Reasoning and Discoverycited by 0
  18. 2026Representation Alignment from a Few Paired Data via Distilled Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilingual Vision-Language Pre-training for the Remote Sensing Domain
  2. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  3. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment
  4. 2026Semi-Supervised Adaptation of Vision-Language Models for Image Classification
  5. 2023Improved baselines for vision-language pre-training
  6. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  7. 2024DiffCLIP: Few-shot Language-driven Multimodal Classifier
  8. 2024RWKV-CLIP: A Robust Vision-Language Representation Learner