mlsanityData to 7 Oct 2026

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

arXiv 2106.16138 ↗First posted 30 Jun 2021cs.CLcited by 72

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
29 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  10. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  11. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  12. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  13. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  14. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  15. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  16. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  17. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  18. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  19. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  20. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  21. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  22. 2020On Learning Universal Representations Across Languagescited by 58
  23. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  24. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  25. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  26. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  27. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  28. 2020Multilingual BERT Post-Pretraining Alignmentcited by 30
  29. 2021Learning to Sample Replacements for ELECTRA Pre-Trainingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 20; 2023: 19; 2024: 18; 2025: 3; 2026: 2.

72 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  3. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  7. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  10. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  11. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  12. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
38 more
  1. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  2. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  3. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  4. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  5. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 40
  6. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  7. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  8. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  9. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  10. 2022QAmeleon: Multilingual QA with Only 5 Examplescited by 25
  11. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  12. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 21
  13. 2022GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminatorcited by 20
  14. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  15. 2023TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processescited by 18
  16. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  17. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  18. 2022GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understandingcited by 14
  19. 2024YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentationcited by 13
  20. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  21. 2024Fairness Definitions in Language Models Explainedcited by 10
  22. 2022Pretraining Text Encoders with Adversarial Mixture of Training Signal Generatorscited by 9
  23. 2022Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretrainingcited by 9
  24. 2023T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classificationcited by 8
  25. 2022Bridging the Gap between Language Models and Cross-Lingual Sequence Labelingcited by 7
  26. 2023Measuring the Instability of Fine-Tuningcited by 7
  27. 2023Machine-Created Universal Language for Cross-lingual Transfercited by 7
  28. 2024Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applicationscited by 7
  29. 2022Short Text Pre-training with Extended Token Classification for E-commerce Query Understandingcited by 5
  30. 2022Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLUcited by 5
  31. 2022Speech separation with large-scale self-supervised learningcited by 5
  32. 2022Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrievalcited by 5
  33. 2023QCRI at SemEval-2023 Task 3: News Genre, Framing and Persuasion Techniques Detection using Multilingual Modelscited by 5
  34. 2023Soft Language Clustering for Multilingual Model Pre-trainingcited by 4
  35. 2024EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learningcited by 4
  36. 2025Explaining Large Language Models with gSMILEcited by 4
  37. 2022'Beach' to 'Bitch': Inadvertent Unsafe Transcription of Kids' Content on YouTubecited by 3
  38. 2022Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusterscited by 3

50 most cited of 72 All 72 (JSON)