mlsanityData to 7 Oct 2026

EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning

arXiv 2310.17233 ↗First posted 26 Oct 2023cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Expressing universal semantics common to all languages is helpful in understanding the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic ``universals'' for any two languages. In this paper, we propose EMMA-X: an EM-like Multilingual pre-training Algorithm, to learn (X)Cross-lingual universals with the aid of excessive multilingual non-parallel data. EMMA-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate EMMA-X, we conduct experiments on XRETE, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that EMMA-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of EMMA-X over advanced models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 60 of its 83 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

60 of 83 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  12. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  13. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  14. 2020Understanding the Behaviour of Contrastive Losscited by 397
  15. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  16. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  17. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  18. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  19. 2019Massively Multilingual Neural Machine Translationcited by 246
  20. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  21. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  22. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  23. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  24. 2020The Multilingual Amazon Reviews Corpuscited by 142
  25. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  26. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  27. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  28. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  29. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  30. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  31. 2020On Learning Universal Representations Across Languagescited by 58
  32. 2020Entity Linking in 100 Languagescited by 52
  33. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  34. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  35. 2019OpenKiwi: An Open Source Framework for Quality Estimationcited by 49
  36. 2020LAReQA: Language-agnostic answer retrieval from a multilingual poolcited by 41
  37. 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
  38. 2022Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positivescited by 37

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 4