mlsanityData to 14 Oct 2026

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger

arXiv 2303.17561 ↗First posted 30 Mar 2023cs.CV cs.AIcited by 56

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 38 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

25 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining29 Apr

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  11. 2013Describing Textures in the Wildcited by 2,125
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
12 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  7. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  8. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  9. 2015YFCC100M: The New Data in Multimedia Researchcited by 235
  10. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 191
  11. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  12. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 25; 2025: 16; 2026: 9.

56 papers.

Sort:

By area:

  1. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  2. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48
  3. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 30
  4. 2023Improved baselines for vision-language pre-trainingcited by 20
  5. 2024Low-Rank Similarity Mining for Multimodal Dataset Distillationcited by 17
  6. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  7. 2025MindSimulator: Exploring Brain Concept Localization via Synthetic FMRIcited by 15
  8. 2024Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrievalcited by 13
  9. 2023CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Losscited by 12
  10. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  11. 2024Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learningcited by 11
  12. 2025Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysiscited by 10
38 more
  1. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  2. 2024mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphscited by 9
  3. 2025SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learningcited by 9
  4. 2024I0T: Embedding Standardization Method Towards Zero Modality Gapcited by 8
  5. 2023Scalable Incomplete Multi-View Clustering with Structure Alignmentcited by 7
  6. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language modelscited by 7
  7. 2024ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Mapcited by 7
  8. 2024MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilizationcited by 6
  9. 2024Multi-modal Relation Distillation for Unified 3D Representation Learningcited by 6
  10. 2025A Vision-Language Foundation Model for Leaf Disease Identificationcited by 6
  11. 2024Training-Free Unsupervised Prompt for Vision-Language Modelscited by 5
  12. 2025Variational Supervised Contrastive Learningcited by 5
  13. 2024Disentangled Noisy Correspondence Learningcited by 4
  14. 2024DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matchingcited by 4
  15. 2025Tables Guide Vision: Learning to See the Heart through Tabular Datacited by 4
  16. 2023Unsupervised Improvement of Audio-Text Cross-Modal Representationscited by 3
  17. 2023LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Modelscited by 3
  18. 2024Cross-sensor self-supervised training and alignment for remote sensingcited by 3
  19. 2024Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution Viewcited by 3
  20. 2024Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Learning with Dense Labelingcited by 3
  21. 2025EcoWikiRS: Learning Ecological Representation of Satellite Images from Weak Supervision with Species Observations and Wikipediacited by 3
  22. 2025Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessmentcited by 3
  23. 2024Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classificationcited by 2
  24. 2025SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRIcited by 2
  25. 2025RLM: A Vision-Language Model Approach for Radar Scene Understandingcited by 2
  26. 2025Learning complete and explainable visual representations from itemized text supervisioncited by 2
  27. 2024Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneitycited by 1
  28. 2024No Captions, No Problem: Captionless 3D-CLIP Alignment with Hard Negatives via CLIP Knowledge and LLMscited by 1
  29. 2024Order-Preserving Dimension Reduction for Multimodal Semantic Embeddingcited by 1
  30. 2025Unlearning the Noisy Correspondence Makes CLIP More Robustcited by 1
  31. 2025FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attentioncited by 1
  32. 2026MindCine: Multimodal EEG-to-Video Reconstruction with Large-Scale Pretrained Modelscited by 1
  33. 2026Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrievalcited by 1
  34. 2026Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representationscited by 1
  35. 2024RESTORE: Towards Feature Shift for Vision-Language Prompt Learningcited by 0
  36. 2025Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captionscited by 0
  37. 2025Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Modelscited by 0
  38. 2025Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learningcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
  2. 2022Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment
  3. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
  4. 2025Modest-Align: Data-Efficient Alignment for Vision-Language Models
  5. 2026BiCLIP: Domain Canonicalization via Structured Geometric Transformation
  6. 2024CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training
  7. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  8. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer