mlsanityData to 7 Oct 2026

Language-agnostic BERT Sentence Embedding

arXiv 2007.01852 ↗First posted 3 Jul 2020cs.CLcited by 621

Not on the map. Why?

Abstract (arXiv)

While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.

Key facts. Cited by 621 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 69; 2022: 78; 2023: 105; 2024: 138; 2025: 120; 2026: 111.

621 papers.

Sort:

By area:

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  3. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  5. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  6. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  7. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  8. 2024Generative Representational Instruction Tuningcited by 227
  9. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  10. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  11. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
  12. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
38 more
  1. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  2. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  3. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  4. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  5. 2021Survey of Low-Resource Machine Translationcited by 92
  6. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  7. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  8. 2024GRAG: Graph Retrieval-Augmented Generationcited by 77
  9. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  10. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  11. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  12. 2022MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Datasetcited by 59
  13. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  14. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  15. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  16. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  17. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  18. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  19. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  20. 2021Claim Matching Beyond English to Scale Global Fact-Checkingcited by 42
  21. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  22. 2022A Survey on Cross-Lingual Summarizationcited by 38
  23. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  24. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  25. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
  26. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  27. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  28. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  29. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  30. 2022SelfKG: Self-Supervised Entity Alignment in Knowledge Graphscited by 32
  31. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  32. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  33. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarkscited by 32
  34. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  35. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 31
  36. 2023MasakhaNEWS: News Topic Classification for African languagescited by 30
  37. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 30
  38. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29

50 most cited of 621 All 621 (JSON)