EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning
Not on the map. Why?
Abstract (arXiv)
Massively multilingual sentence representation models, e.g., LASER, SBERT-distill, and LaBSE, help significantly improve cross-lingual downstream tasks. However, the use of a large amount of data or inefficient model architectures results in heavy computation to train a new model according to our preferred languages and domains. To resolve this issue, we introduce efficient and effective massively multilingual sentence embedding (EMS), using cross-lingual token-level reconstruction (XTR) and sentence-level contrastive learning as training objectives. Compared with related studies, the proposed model can be efficiently trained using significantly fewer parallel sentences and GPU computation resources. Empirical results showed that the proposed model significantly yields better or comparable results with regard to cross-lingual sentence retrieval, zero-shot cross-lingual genre classification, and sentiment classification. Ablative analyses demonstrated the efficiency and effectiveness of each component of the proposed model. We release the codes for model training and the EMS pre-trained sentence embedding model, which supports 62 languages ( https://github.com/Mao-KU/EMS ).
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 56 of its 92 references that are in the map. First posted on arXiv 31 May 2022.
Builds on
56 of 92 references.
References · 56
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
38 more
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
- 2020Language-agnostic BERT Sentence Embeddingcited by 627
- 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
- 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
- 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
- 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
- 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
- 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
- 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
- 2018An efficient framework for learning sentence representationscited by 198
- 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
- 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
- 2020The Multilingual Amazon Reviews Corpuscited by 143
- 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
- 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
- 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
- 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
- 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
- 2020On Learning Universal Representations Across Languagescited by 58
- 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
- 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
- 2018Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Modelcited by 46
- 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
- 2018Learning Semantic Textual Similarity from Conversationscited by 41
- 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
- 2015ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairscited by 35
- 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34
- 2019Explicit Cross-lingual Pre-training for Unsupervised Machine Translationcited by 13
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.