mlsanityData to 7 Oct 2026

UniCLIP: Unified Framework for Contrastive Language-Image Pre-training

arXiv 2209.13430 ↗First posted 27 Sep 2022cs.CV cs.LGcited by 54

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-training vision-language models with contrastive objectives has shown promising results that are both scalable to large uncurated datasets and transferable to many downstream applications. Some following works have targeted to improve data efficiency by adding self-supervision terms, but inter-domain (image-text) contrastive loss and intra-domain (image-image) contrastive loss are defined on individual spaces in those works, so many feasible combinations of supervision are overlooked. To overcome this issue, we propose UniCLIP, a Unified framework for Contrastive Language-Image Pre-training. UniCLIP integrates the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space. The discrepancies that occur when integrating contrastive loss between different domains are resolved by the three key components of UniCLIP: (1) augmentation-aware feature embedding, (2) MP-NCE loss, and (3) domain dependent similarity measure. UniCLIP outperforms previous vision-language pre-training methods on various single- and multi-modality downstream tasks. In our experiments, we show that each component that comprises UniCLIP contributes well to the final performance.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 50 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

36 of 50 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
24 more
  1. 2020Supervised Contrastive Learningcited by 2,803
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2016Colorful Image Colorizationcited by 1,113
  15. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  16. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  17. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  18. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  19. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  20. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  21. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  22. 2017Multi-task Self-Supervised Visual Learningcited by 200
  23. 2017Representation Learning by Learning to Countcited by 120
  24. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 16; 2025: 12; 2026: 9.

54 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  3. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  4. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  5. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  6. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  7. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  8. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  9. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  10. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  11. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  12. 2022Non-Contrastive Learning Meets Language-Image Pre-Trainingcited by 25
38 more
  1. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  2. 2024RWKV-CLIP: A Robust Vision-Language Representation Learnercited by 23
  3. 2024Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIPcited by 23
  4. 2023Improved baselines for vision-language pre-trainingcited by 20
  5. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  6. 2022Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoderscited by 15
  7. 2023CILF-CIAE: CLIP-driven Image-Language Fusion for Correcting Inverse Age Estimationcited by 15
  8. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  9. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  10. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  11. 2023Misalign, Contrast then Distill: Rethinking Misalignments in Language-Image Pretrainingcited by 10
  12. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Modelscited by 10
  13. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  14. 2025CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separationcited by 9
  15. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  16. 2023MOFI: Learning Image Representations from Noisy Entity Annotated Imagescited by 7
  17. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language modelscited by 7
  18. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 7
  19. 2024Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignmentcited by 7
  20. 2024MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilizationcited by 6
  21. 2025A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generationcited by 6
  22. 2022Retrieval-based Disentangled Representation Learning with Natural Language Supervisioncited by 5
  23. 2025RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capabilitycited by 4
  24. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillationcited by 4
  25. 2023LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Modelscited by 3
  26. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3
  27. 2023Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objectivecited by 2
  28. 2026MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrievalcited by 2
  29. 2025Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understandingcited by 1
  30. 2025ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interactioncited by 1
  31. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1
  32. 2023ELVIS: Empowering Locality of Vision Language Pre-training with Intra-modal Similaritycited by 0
  33. 2024Mining Open Semantics from CLIP: A Relation Transition Perspective for Few-Shot Learningcited by 0
  34. 2025Impact of Language Guidance: A Reproducibility Studycited by 0
  35. 2025Leveraging Geometric Visual Illusions as Perceptual Inductive Biases for Vision Modelscited by 0
  36. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0
  37. 2026Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Modelcited by 0
  38. 2026Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representationcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models
  2. 2023ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training
  4. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  5. 2025TULIP: Towards Unified Language-Image Pretraining
  6. 2022Unifying Vision-Language Representation Space with Single-tower Transformer
  7. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  8. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning