mlsanityData to 7 Oct 2026

Multimodal Contrastive Training for Visual Representation Learning

arXiv 2104.12836 ↗First posted 26 Apr 2021cs.CVcited by 122

Not on the map. Why?

Abstract (arXiv)

We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of 55.3% on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

48 of 62 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2017Mask R-CNNcited by 7,259
  12. 2016Densely Connected Convolutional Networkscited by 6,691
36 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  4. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2016Colorful Image Colorizationcited by 1,113
  16. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  17. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  20. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  21. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  23. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  24. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  25. 2018Scaling Neural Machine Translationcited by 204
  26. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  27. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  28. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  29. 2020Learning Visual Representations with Caption Annotationscited by 123
  30. 2016Learning Visual N-Grams from Web Datacited by 95
  31. 2017MegDet: A Large Mini-Batch Object Detectorcited by 66
  32. 2017Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Modelscited by 53
  33. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43
  34. 2016An Empirical Study of Language CNN for Image Captioningcited by 15
  35. 2016Salient Object Subitizingcited by 13
  36. 2018Weakly Supervised Deep Image Hashing through Tag Embeddingscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 28; 2023: 20; 2024: 33; 2025: 20; 2026: 11.

122 papers.

Sort:

By area:

  1. 2021Extract Free Dense Labels from CLIPcited by 580
  2. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  3. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2023Improving CLIP Training with Language Rewritescited by 216
  6. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  7. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  8. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  9. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  10. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  11. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  12. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
38 more
  1. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  2. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  3. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  4. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  5. 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
  6. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  7. 2024What to align in multimodal contrastive learning?cited by 41
  8. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 38
  9. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 35
  10. 2024Time Weaver: A Conditional Time Series Generation Modelcited by 35
  11. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 31
  12. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 30
  13. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  14. 2023IMF: Interactive Multimodal Fusion Model for Link Predictioncited by 28
  15. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  16. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  17. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  18. 2022CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detectioncited by 21
  19. 2022COCOA: Cross Modality Contrastive Learning for Sensor Datacited by 21
  20. 2022VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignmentcited by 20
  21. 2021Learning to Align Sequential Actions in the Wildcited by 19
  22. 2023Lego: Learning to Disentangle and Invert Personalized Concepts Beyond Object Appearance in Text-to-Image Diffusion Modelscited by 19
  23. 2024VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptionscited by 18
  24. 2022VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classificationcited by 14
  25. 2023Test-Time Distribution Normalization for Contrastively Learned Vision-language Modelscited by 14
  26. 2024An Information Criterion for Controlled Disentanglement of Multimodal Datacited by 14
  27. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  28. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  29. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  30. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Modelscited by 13
  31. 2022SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learningcited by 12
  32. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  33. 2025To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performancecited by 12
  34. 2022mulEEG: A Multi-View Representation Learning on EEG Signalscited by 11
  35. 2023Cross-Modal Retrieval for Motion and Text via DropTriple Losscited by 11
  36. 2022Deep Unsupervised Contrastive Hashing for Large-Scale Cross-Modal Text-Image Retrieval in Remote Sensingcited by 10
  37. 2022e-CLIP: Large-Scale Vision-Language Representation Learning in E-commercecited by 10
  38. 2023A Multi-Modal Contrastive Diffusion Model for Therapeutic Peptide Generationcited by 10

50 most cited of 122 All 122 (JSON)