mlsanityData to 7 Oct 2026

English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings

arXiv 2211.06127 ↗First posted 11 Nov 2022cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences. In this work, we propose mSimCSE, which extends SimCSE to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data. In unsupervised and weakly supervised settings, mSimCSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low-resource languages and multilingual STS. The performance can be further enhanced when cross-lingual NLI data is available. Our code is publicly available at https://github.com/yaushian/mSimCSE.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

41 of 49 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  11. 2019How multilingual is Multilingual BERT?cited by 642
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 621
29 more
  1. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  2. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  3. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  4. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  5. 2015Skip-Thought Vectorscited by 321
  6. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  8. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  9. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  10. 2018An efficient framework for learning sentence representationscited by 197
  11. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  12. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  13. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  14. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  15. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  16. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  17. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  18. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  19. 2020On Learning Universal Representations Across Languagescited by 58
  20. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  21. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  22. 2018Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Modelcited by 46
  23. 2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
  24. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  25. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  26. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  27. 2019Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NERcited by 25
  28. 2021Paraphrastic Representations at Scalecited by 19
  29. 2022MuCoT: Multilingual Contrastive Training for Question-Answering in Low-resource Languagescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 4; 2026: 4.

26 papers.

Sort:

By area:

  1. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  2. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  3. 2024Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignmentcited by 13
  4. 2023Language Models are Universal Embedderscited by 11
  5. 2023Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embeddingcited by 9
  6. 2024Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applicationscited by 7
  7. 2024LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddingscited by 7
  8. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  9. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  10. 2024Survey in Characterizing Semantic Changecited by 5
  11. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  12. 2023PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classificationcited by 4
14 more
  1. 2024Large Language Model Augmented Exercise Retrieval for Personalized Language Learningcited by 4
  2. 2024News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendationcited by 4
  3. 2024Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervisioncited by 3
  4. 2024Preserving Multilingual Quality While Tuning Query Encoder on English Onlycited by 3
  5. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignmentcited by 2
  6. 2025Translation in the Wildcited by 2
  7. 2026CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-trainingcited by 2
  8. 2024Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similaritycited by 1
  9. 2024Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraintcited by 1
  10. 2023OpenMSD: Towards Multilingual Scientific Documents Similarity Measurementcited by 0
  11. 2025UniBERT: Adversarial Training for Language-Universal Representationscited by 0
  12. 2026ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Taskscited by 0
  13. 2026LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Editscited by 0
  14. 2026Zero-shot Dependency Parsing with Unsupervised Cross-Lingual Bootstrappingcited by 0