mlsanityData to 14 Oct 2026

MCSE: Multimodal Contrastive Learning of Sentence Embeddings

arXiv 2204.10931 ↗First posted 22 Apr 2022cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Learning semantically meaningful sentence embeddings is an open problem in natural language processing. In this work, we propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. Through experiments on a variety of semantic textual similarity tasks, we demonstrate that our approach consistently improves the performance across various datasets and pre-trained encoders. In particular, combining a small amount of multimodal data with a large text-only corpus, we improve the state-of-the-art average Spearman's correlation by 1.7%. By analyzing the properties of the textual embedding space, we show that our model excels in aligning semantically similar sentences, providing an explanation for its improved performance.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 26 of its 49 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

26 of 49 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  12. 2015Skip-Thought Vectorscited by 322
14 more
  1. 2020Experience Grounds Languagecited by 242
  2. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  3. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  4. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  5. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 94
  6. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  7. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  8. 2020Visually Grounded Compound PCFGscited by 34
  9. 2015Combining Language and Vision with a Multimodal Skip-gram Modelcited by 28
  10. 2017Learning Visually Grounded Sentence Representationscited by 24
  11. 2021VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfercited by 22
  12. 2019A Bilingual Generative Transformer for Semantic Sentence Embeddingcited by 16
  13. 2020Incorporating Visual Semantics into Sentence Representations within a Grounded Spacecited by 15
  14. 2017Learning Multi-Modal Word Representation Grounded in Visual Contextcited by 7

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 11; 2024: 5; 2025: 3.

22 papers.

Sort:

By area:

  1. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 33
  2. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
  3. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  4. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  5. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  6. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 12
  7. 2022Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddingscited by 10
  8. 2024I0T: Embedding Standardization Method Towards Zero Modality Gapcited by 8
  9. 2023FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedbackcited by 7
  10. 2023Efficient Domain Adaptation of Sentence Embeddings Using Adapterscited by 6
  11. 2023Medical Question Summarization with Entity-driven Contrastive Learningcited by 5
  12. 2024AAdaM at SemEval-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatednesscited by 4
10 more
  1. 2025Experiential Semantic Information and Brain Alignment: Are Multimodal Models Better than Language Models?cited by 4
  2. 2023DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animationcited by 2
  3. 2024SetCSE: Set Operations using Contrastive Learning of Sentence Embeddingscited by 2
  4. 2025DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learningcited by 2
  5. 2024Multimodal Fusion and Coherence Modeling for Video Topic Segmentationcited by 1
  6. 2025Improving Multimodal Contrastive Learning of Sentence Embeddings with Object-Phrase Alignmentcited by 1
  7. 2022Can Offline Reinforcement Learning Help Natural Language Understanding?cited by 0
  8. 2023D2CSE: Difference-aware Deep continuous prompts for Contrastive Sentence Embeddingscited by 0
  9. 2023Generate to Understand for Representationcited by 0
  10. 2023On the Language Encoder of Contrastive Cross-modal Modelscited by 0