mlsanityData to 7 Oct 2026

Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining

arXiv 2105.10419 ↗First posted 21 May 2021cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 21 May 2021.

Builds on

26 of 38 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2019How multilingual is Multilingual BERT?cited by 642
  6. 2017Word Translation Without Parallel Datacited by 480
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  9. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  10. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  11. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  12. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
14 more
  1. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  2. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  3. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  4. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 69
  5. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  6. 2019How Language-Neutral is Multilingual BERT?cited by 51
  7. 2019Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsingcited by 46
  8. 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
  9. 2019Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spacescited by 35
  10. 2018Filtering and Mining Parallel Data in a Joint Multilingual Spacecited by 32
  11. 2019Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?cited by 31
  12. 2019Universal Text Representation from BERT: An Empirical Studycited by 28
  13. 2019Analyzing the Limitations of Cross-lingual Word Embedding Mappingscited by 18
  14. 2017An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identificationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 1; 2025: 0; 2026: 1.

9 papers.

  1. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  2. 2022EASE: Entity-Aware Contrastive Learning of Sentence Embeddingcited by 21
  3. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  4. 2023Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embeddingcited by 9
  5. 2022Combining Static and Contextualised Multilingual Embeddingscited by 8
  6. 2023EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learningcited by 1
  7. 2022Finetuning a Kalaallisut-English machine translation system using web-crawled datacited by 0
  8. 2023Boosting Unsupervised Machine Translation with Pseudo-Parallel Datacited by 0
  9. 2026Almost Clinical: Linguistic properties of synthetic electronic health recordscited by 0