# English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings

arXiv: https://arxiv.org/abs/2211.06127 · HTML: https://mlsanity.com/paper/2211.06127/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)

## Built on by (most cited first, 26 of 26)

- [AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors](https://mlsanity.com/paper/2502.12191/) (2025)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment](https://mlsanity.com/paper/2404.02490/) (2024)
- [Language Models are Universal Embedders](https://mlsanity.com/paper/2310.08232/) (2023)
- [Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding](https://mlsanity.com/paper/2309.08929/) (2023)
- [Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications](https://mlsanity.com/paper/2403.17528/) (2024)
- [LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings](https://mlsanity.com/paper/2412.03331/) (2024)
- [How Transliterations Improve Crosslingual Alignment](https://mlsanity.com/paper/2409.17326/) (2024)
- [Analysis of Multi-Source Language Training in Cross-Lingual Transfer](https://mlsanity.com/paper/2402.13562/) (2024)
- [Survey in Characterizing Semantic Change](https://mlsanity.com/paper/2402.19088/) (2024)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification](https://mlsanity.com/paper/2305.14963/) (2023)
- [Large Language Model Augmented Exercise Retrieval for Personalized Language Learning](https://mlsanity.com/paper/2402.16877/) (2024)
- [News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation](https://mlsanity.com/paper/2406.12634/) (2024)
- [Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervision](https://mlsanity.com/paper/2402.16508/) (2024)
- [Preserving Multilingual Quality While Tuning Query Encoder on English Only](https://mlsanity.com/paper/2407.00923/) (2024)
- [CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training](https://mlsanity.com/paper/2604.05821/) (2026)
- [Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment](https://mlsanity.com/paper/2407.14878/) (2024)
- [Translation in the Wild](https://mlsanity.com/paper/2505.23548/) (2025)
- [Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity](https://mlsanity.com/paper/2403.05257/) (2024)
- [Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint](https://mlsanity.com/paper/2409.15664/) (2024)
- [OpenMSD: Towards Multilingual Scientific Documents Similarity Measurement](https://mlsanity.com/paper/2309.10539/) (2023)
- [UniBERT: Adversarial Training for Language-Universal Representations](https://mlsanity.com/paper/2503.12608/) (2025)
- [ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks](https://mlsanity.com/paper/2603.21084/) (2026)
- [LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits](https://mlsanity.com/paper/2609.18720/) (2026)
- [Zero-shot Dependency Parsing with Unsupervised Cross-Lingual Bootstrapping](https://mlsanity.com/paper/2609.37883/) (2026)

## Abstract (arXiv)

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences. In this work, we propose mSimCSE, which extends SimCSE to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data. In unsupervised and weakly supervised settings, mSimCSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low-resource languages and multilingual STS. The performance can be further enhanced when cross-lingual NLI data is available. Our code is publicly available at https://github.com/yaushian/mSimCSE.