mlsanityData to 7 Oct 2026

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

arXiv 2210.06432 ↗First posted 8 Oct 2022cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments. Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods. Our code are available at https://github.com/caskcsg/sentemb/tree/main/InfoCSE.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 8 Oct 2022.

Builds on

24 of 43 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2014Distributed Representations of Sentences and Documentscited by 677
  7. 2015Skip-Thought Vectorscited by 321
  8. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  9. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  10. 2018An efficient framework for learning sentence representationscited by 197
  11. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  12. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
12 more
  1. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  2. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  3. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  4. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  5. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  6. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  7. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  8. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  9. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  10. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  11. 2021Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 14
  12. 2021Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decodercited by 7

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 16; 2024: 2; 2025: 5.

25 papers.

Sort:

By area:

  1. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  2. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  3. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  4. 2023Composition-contrastive Learning for Sentence Embeddingscited by 14
  5. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  6. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7
  7. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  8. 2023Length is a Curse and a Blessing for Document-level Semanticscited by 6
  9. 2023Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learningcited by 3
  10. 2023Some Like It Small: Czech Semantic Embedding Models for Industry Applicationscited by 3
  11. 2025TreeHop: Efficient Embedding-Level Query Rewritercited by 3
  12. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
13 more
  1. 2023Weighted Sampling for Masked Language Modelingcited by 2
  2. 2023Non-contrastive sentence representations via self-supervisioncited by 2
  3. 2025Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Modelscited by 2
  4. 2023Adapting Sentence Transformers for the Aviation Domaincited by 1
  5. 2023HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddingscited by 1
  6. 2023EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learningcited by 1
  7. 2023Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gendercited by 1
  8. 2023AdaptiveRec: Adaptively Construct Pairs for Contrastive Learning in Sequential Recommendationcited by 0
  9. 2023Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Modelcited by 0
  10. 2023Semi-supervised News Discourse Profiling with Contrastive Learningcited by 0
  11. 2024Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysiscited by 0
  12. 2025TNCSE: Tensor's Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddingscited by 0
  13. 2025MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learningcited by 0