# InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

arXiv: https://arxiv.org/abs/2210.06432 · HTML: https://mlsanity.com/paper/2210.06432/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 8 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04380/) (2021)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [Smoothed Contrastive Learning for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04321/) (2021)
- [Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder](https://mlsanity.com/paper/2102.09206/) (2021)

## Built on by (most cited first, 26 of 26)

- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank](https://mlsanity.com/paper/2305.16726/) (2023)
- [On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning](https://mlsanity.com/paper/2212.09170/) (2022)
- [Composition-contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2307.07380/) (2023)
- [A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond](https://mlsanity.com/paper/2305.12641/) (2023)
- [miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings](https://mlsanity.com/paper/2211.04928/) (2022)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [Length is a Curse and a Blessing for Document-level Semantics](https://mlsanity.com/paper/2310.16193/) (2023)
- [Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning](https://mlsanity.com/paper/2310.10962/) (2023)
- [Some Like It Small: Czech Semantic Embedding Models for Industry Applications](https://mlsanity.com/paper/2311.13921/) (2023)
- [TreeHop: Efficient Embedding-Level Query Rewriter](https://mlsanity.com/paper/2504.20114/) (2025)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Weighted Sampling for Masked Language Modeling](https://mlsanity.com/paper/2302.14225/) (2023)
- [Non-contrastive sentence representations via self-supervision](https://mlsanity.com/paper/2310.17690/) (2023)
- [Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models](https://mlsanity.com/paper/2502.13656/) (2025)
- [Adapting Sentence Transformers for the Aviation Domain](https://mlsanity.com/paper/2305.09556/) (2023)
- [HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings](https://mlsanity.com/paper/2310.09720/) (2023)
- [EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning](https://mlsanity.com/paper/2310.17233/) (2023)
- [Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gender](https://mlsanity.com/paper/2310.19130/) (2023)
- [AdaptiveRec: Adaptively Construct Pairs for Contrastive Learning in Sequential Recommendation](https://mlsanity.com/paper/2307.05469/) (2023)
- [Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model](https://mlsanity.com/paper/2309.06453/) (2023)
- [Semi-supervised News Discourse Profiling with Contrastive Learning](https://mlsanity.com/paper/2309.11692/) (2023)
- [Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysis](https://mlsanity.com/paper/2405.11297/) (2024)
- [TNCSE: Tensor's Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2503.12739/) (2025)
- [MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning](https://mlsanity.com/paper/2510.16797/) (2025)
- [CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking](https://mlsanity.com/paper/2610.02460/) (2026)

## Abstract (arXiv)

Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments. Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods. Our code are available at https://github.com/caskcsg/sentemb/tree/main/InfoCSE.