mlsanityData to 7 Oct 2026

CyCLIP: Cyclic Contrastive Language-Image Pretraining

arXiv 2205.14459 ↗First posted 28 May 2022cs.CV cs.LGcited by 127

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness. Such models typically require joint reasoning in the image and text representation spaces for downstream inference tasks. Contrary to prior beliefs, we demonstrate that the image and text representations learned via a standard contrastive objective are not interchangeable and can lead to inconsistent downstream predictions. To mitigate this issue, we formalize consistency and propose CyCLIP, a framework for contrastive representation learning that explicitly optimizes for the learned representations to be geometrically consistent in the image and text space. In particular, we show that consistent representations can be learned by explicitly symmetrizing (a) the similarity between the two mismatched image-text pairs (cross-modal consistency); and (b) the similarity between the image-image pair and the text-text pair (in-modal consistency). Empirically, we show that the improved consistency in CyCLIP translates to significant gains over CLIP, with gains ranging from 10%-24% for zero-shot classification accuracy on standard benchmarks (CIFAR-10, CIFAR-100, ImageNet1K) and 10%-27% for robustness to various natural distribution shifts. The code is available at https://github.com/goel-shashank/CyCLIP.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 49 of its 77 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

49 of 77 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
37 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  13. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  15. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  16. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  17. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  18. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  19. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  20. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  21. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  22. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  23. 2017DualGAN: Unsupervised Dual Learning for Image-to-Image Translationcited by 310
  24. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  25. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  26. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  27. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  28. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  29. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  30. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  31. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  32. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  33. 2022Multi-modal Alignment using Representation Codebookcited by 63
  34. 2021Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysiscited by 41
  35. 2019AlignFlow: Cycle Consistent Learning from Multiple Domains via Normalizing Flowscited by 33
  36. 2017See, Hear, and Read: Deep Aligned Representationscited by 32
  37. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 42; 2024: 42; 2025: 19; 2026: 15.

127 papers.

Sort:

By area:

  1. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  2. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  4. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  7. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  8. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  9. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  10. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  11. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  12. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
38 more
  1. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  2. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 62
  3. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  4. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  5. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Strongercited by 56
  6. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  7. 2023EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methodscited by 52
  8. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  9. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  10. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  11. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  12. 2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
  13. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  14. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  15. 2023An Empirical Study of CLIP for Text-based Person Searchcited by 38
  16. 2023Distribution-Aware Prompt Tuning for Vision-Language Modelscited by 35
  17. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  18. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  19. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attackscited by 31
  20. 2022Graph Matching with Bi-level Noisy Correspondencecited by 30
  21. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  22. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  23. 2024It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gapcited by 26
  24. 2023Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualizationcited by 25
  25. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
  26. 2025LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulationcited by 24
  27. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  28. 2024Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIPcited by 23
  29. 2023Cross-modal Active Complementary Learning with Self-refining Correspondencecited by 22
  30. 2024Visual Neural Decoding via Improved Visual-EEG Semantic Consistencycited by 21
  31. 2022ConStruct-VL: Data-Free Continual Structured VL Concepts Learningcited by 20
  32. 2022ADIR: Adaptive Diffusion for Image Reconstructioncited by 20
  33. 2023Improved baselines for vision-language pre-trainingcited by 20
  34. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attackscited by 19
  35. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  36. 2023Improved Visual Grounding through Self-Consistent Explanationscited by 17
  37. 2022Stochastic Constrained DRO with a Complexity Independent of Sample Sizecited by 16
  38. 2023Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotationscited by 16

50 most cited of 127 All 127 (JSON)