mlsanityData to 7 Oct 2026

Cross-lingual Language Model Pretraining

arXiv 1901.07291 ↗First posted 22 Jan 2019cs.CLcited by 1,088

Not on the map. Why?

Abstract (arXiv)

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

Key facts. Cited by 1,088 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 343; 2022: 244; 2023: 198; 2024: 177; 2025: 83; 2026: 43.

1,088 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  11. 2024Large Language Models: A Surveycited by 506
  12. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
38 more
  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  2. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  4. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  5. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  6. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  7. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  8. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  9. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  10. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  12. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  13. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  14. 2021Anticipative Video Transformercited by 197
  15. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
  16. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  17. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  18. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  19. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  20. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  21. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  22. 2021The Values Encoded in Machine Learning Researchcited by 172
  23. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  24. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  25. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  26. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  27. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  28. 2021Unsupervised Speech Recognitioncited by 147
  29. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  30. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  31. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  32. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  33. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  34. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  35. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  36. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  37. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  38. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 130

50 most cited of 1,088 All 1,088 (JSON)