mlsanityData to 7 Oct 2026

Unsupervised Cross-lingual Representation Learning at Scale

arXiv 1911.02116 ↗First posted 5 Nov 2019cs.CLcited by 3,907

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.

Key facts. Cited by 3,907 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 532; 2022: 662; 2023: 738; 2024: 829; 2025: 628; 2026: 517.

3,907 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
38 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  3. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  4. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  9. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  10. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  11. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  13. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  14. 2023Scaling Data-Constrained Language Modelscited by 391
  15. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  16. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  17. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  18. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  19. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  20. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  21. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  22. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  23. 2023Improving Text Embeddings with Large Language Modelscited by 295
  24. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  25. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  26. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  27. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  28. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  29. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  30. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  31. 2021Hash Layers For Large Sparse Modelscited by 236
  32. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  33. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  34. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  35. 2022Unified Structure Generation for Universal Information Extractioncited by 220
  36. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  37. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  38. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205

50 most cited of 3,907 All 3,907 (JSON)