mlsanityData to 7 Oct 2026

Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast

arXiv 2109.00253 ↗First posted 1 Sep 2021cs.CL cs.IRcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolingual) can be computed with a simple dot product. Pre-trained language models are fine-tuned with the translation ranking task. Existing work (Feng et al., 2020) uses sentences within the same batch as negatives, which can suffer from the issue of easy negatives. We adapt MoCo (He et al., 2020) to further improve the quality of alignment. As the experimental results show, the sentence representations produced by our model achieve the new state-of-the-art on several tasks, including Tatoeba en-zh similarity search (Artetxe and Schwenk, 2019b), BUCC en-zh bitext mining, and semantic textual similarity on 7 datasets.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

32 of 47 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
20 more
  1. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  2. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  3. 2017Word Translation Without Parallel Datacited by 480
  4. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  5. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  8. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  9. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  10. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  11. 2019Massively Multilingual Neural Machine Translationcited by 246
  12. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  13. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  14. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  15. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  16. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  17. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  18. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  19. 2018Learning Semantic Textual Similarity from Conversationscited by 41
  20. 2019Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Frameworkcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 2; 2024: 0; 2025: 0; 2026: 2.

7 papers.

  1. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  2. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  3. 2022Multi-Level Contrastive Learning for Cross-Lingual Alignmentcited by 4
  4. 2021DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddingscited by 2
  5. 2023EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learningcited by 1
  6. 2026Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performancecited by 1
  7. 2026AlphaFree: Recommendation Free from Users, IDs, and GNNscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining
  2. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment
  3. 2022Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment
  4. 2023Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
  5. 2024Exploring Alignment in Shared Cross-lingual Spaces
  6. 2025Benchmarking Cross-Lingual Semantic Alignment in Multilingual Embeddings
  7. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  8. 2024Understanding Cross-Lingual Alignment -- A Survey