mlsanityData to 7 Oct 2026

Vision-Language Pre-Training with Triple Contrastive Learning

arXiv 2202.10401 ↗First posted 21 Feb 2022cs.CVcited by 212

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text. However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations. For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by. This problem can get even worse when the pre-training data is noisy. In this paper, we propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning. To take advantage of localized and structural information from image and text input, TCL further maximizes the average MI between local regions of image/text and their global summary. To the best of our knowledge, ours is the first work that takes into account local structure information for multi-modality representation learning. Experimental evaluations show that our approach is competitive and achieves the new state of the art on various common down-stream vision-language tasks such as image-text retrieval and visual question answering.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 50 references that are in the map. First posted on arXiv 21 Feb 2022.

Builds on

36 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022A Survey of Vision-Language Pre-Trained Models18 Feb
  2. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
21 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  14. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  15. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  16. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  17. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  18. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  19. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  20. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  21. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 77; 2024: 50; 2025: 35; 2026: 16.

212 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  4. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  7. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  8. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  9. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  10. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  11. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  12. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
38 more
  1. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  2. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 106
  3. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 98
  4. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  5. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  6. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  7. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  8. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  9. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  10. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  11. 2022Multi-modal Alignment using Representation Codebookcited by 63
  12. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  13. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  14. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  15. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  16. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  17. 2023RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Searchcited by 53
  18. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  19. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  20. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  21. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  22. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  23. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41
  24. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  25. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
  26. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 39
  27. 2023Decoupling Common and Unique Representations for Multimodal Self-supervised Learningcited by 36
  28. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
  29. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  30. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 35
  31. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Modelscited by 31
  32. 2022Geodesic Multi-Modal Mixup for Robust Fine-Tuningcited by 29
  33. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  34. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  35. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Modelscited by 28
  36. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  37. 2023All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignmentcited by 27
  38. 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detectioncited by 26

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Contrastive Vision-Language Pre-training with Limited Resources
  2. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training
  3. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
  4. 2023TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training
  5. 2023Contrastive Vision-Language Alignment Makes Efficient Instruction Learner
  6. 2023Improved baselines for vision-language pre-training
  7. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  8. 2023CAVL: Learning Contrastive and Adaptive Representations of Vision and Language