mlsanityData to 7 Oct 2026

A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

arXiv 2006.06202 ↗First posted 11 Jun 2020cs.CLcited by 131

Not on the map. Why?

Abstract (arXiv)

We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 20; 2023: 22; 2024: 32; 2025: 20; 2026: 14.

131 papers.

Sort:

By area:

  1. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  2. 2023Scaling Data-Constrained Language Modelscited by 391
  3. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  4. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  5. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  6. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  7. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  8. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  9. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  10. 2021Distributed Deep Learning in Open Collaborationscited by 58
  11. 2021Pre-Training BERT on Arabic Tweets: Practical Considerationscited by 39
  12. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
38 more
  1. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  2. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  3. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  4. 2021AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application Withcited by 28
  5. 2023Typhoon: Thai Large Language Modelscited by 28
  6. 2022IndicXNLI: Evaluating Multilingual Inference for Indian Languagescited by 26
  7. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  8. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  9. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  10. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  11. 2023Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Trainingcited by 23
  12. 2022Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Datacited by 22
  13. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  14. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  15. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 20
  16. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  17. 2021Machine Translation into Low-resource Language Varietiescited by 19
  18. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 19
  19. 2024Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruningcited by 19
  20. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  21. 2021Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extractioncited by 16
  22. 2021MFAQ: a Multilingual FAQ Datasetcited by 16
  23. 2024SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languagescited by 16
  24. 2024A Law of Next-Token Prediction in Large Language Modelscited by 15
  25. 2025TigerLLM - A Family of Bangla Large Language Modelscited by 15
  26. 2021HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognitioncited by 14
  27. 2023PersianLLaMA: Towards Building First Persian Large Language Modelcited by 14
  28. 2022Breaking Character: Are Subwords Good Enough for MRLs After All?cited by 13
  29. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 13
  30. 2021Adapting Monolingual Models: Data can be Scarce when Language Similarity is Highcited by 12
  31. 2021ParaShoot: A Hebrew Question Answering Datasetcited by 12
  32. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  33. 2023HeRo: RoBERTa and Longformer Hebrew Language Modelscited by 12
  34. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  35. 2024Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model promptingcited by 12
  36. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  37. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  38. 2022You Are What You Write: Preserving Privacy in the Era of Large Language Modelscited by 11

50 most cited of 131 All 131 (JSON)