# Universal Sentence Representation Learning with Conditional Masked Language Model

arXiv: https://arxiv.org/abs/2012.14388 · HTML: https://mlsanity.com/paper/2012.14388/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2020.

## Built on by (most cited first, 30 of 34)

- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [EASE: Entity-Aware Contrastive Learning of Sentence Embedding](https://mlsanity.com/paper/2205.04260/) (2022)
- [Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision](https://mlsanity.com/paper/2206.14719/) (2022)
- [A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations](https://mlsanity.com/paper/2109.04727/) (2021)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [German Text Embedding Clustering Benchmark](https://mlsanity.com/paper/2401.02709/) (2024)
- [Sentence Representation Learning with Generative Objective rather than Contrastive Objective](https://mlsanity.com/paper/2210.08474/) (2022)
- [A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond](https://mlsanity.com/paper/2305.12641/) (2023)
- [Language Models are Universal Embedders](https://mlsanity.com/paper/2310.08232/) (2023)
- [Dual-Alignment Pre-training for Cross-lingual Sentence Embedding](https://mlsanity.com/paper/2305.09148/) (2023)
- [Autonomous LLM-Enhanced Adversarial Attack for Text-to-Motion](https://mlsanity.com/paper/2408.00352/) (2024)
- [Global Contrastive Batch Sampling via Optimization on Sample Permutations](https://mlsanity.com/paper/2210.12874/) (2022)
- [DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning](https://mlsanity.com/paper/2401.13621/) (2024)
- [Learning Job Title Representation from Job Description Aggregation Network](https://mlsanity.com/paper/2406.08055/) (2024)
- [On the Role of Discreteness in Diffusion LLMs](https://mlsanity.com/paper/2512.22630/) (2025)
- [CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning](https://mlsanity.com/paper/2506.17345/) (2025)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Easy Guided Decoding in Providing Suggestions for Interactive Machine Translation](https://mlsanity.com/paper/2211.07093/) (2022)
- [Auto-MLM: Improved Contrastive Learning for Self-supervised Multi-lingual Knowledge Retrieval](https://mlsanity.com/paper/2203.16187/) (2022)
- [EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning](https://mlsanity.com/paper/2205.15744/) (2022)
- [A Commonsense-Infused Language-Agnostic Learning Framework for Enhancing Prediction of Political Polarity in Multilingual News Headlines](https://mlsanity.com/paper/2212.00298/) (2022)
- [RepCL: Exploring Effective Representation for Continual Text Classification](https://mlsanity.com/paper/2305.07289/) (2023)
- [Surge Routing: Event-informed Multiagent Reinforcement Learning for Autonomous Rideshare](https://mlsanity.com/paper/2307.02637/) (2023)
- [Improving Cross-lingual Representation for Semantic Retrieval with Code-switching](https://mlsanity.com/paper/2403.01364/) (2024)
- [Space Decomposition for Sentence Embedding](https://mlsanity.com/paper/2406.03125/) (2024)
- [Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning](https://mlsanity.com/paper/2204.09358/) (2022)
- [D2CSE: Difference-aware Deep continuous prompts for Contrastive Sentence Embeddings](https://mlsanity.com/paper/2304.08991/) (2023)

## Abstract (arXiv)

This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of-the-art performance on SentEval, even outperforming models learned using supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin, e.g. 10% improvement upon baseline models on cross-lingual semantic search. We explore the same language bias of the learned representations, and propose a simple, post-training and model agnostic approach to remove the language identifying information from the representation while still retaining sentence semantics.