mlsanityData to 7 Oct 2026

Multilingual Open Text Release 1: Public Domain News in 44 Languages

arXiv 2201.05609 ↗First posted 14 Jan 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

We present Multilingual Open Text (MOT), a new multilingual corpus containing text in 44 languages, many of which have limited existing text resources for natural language processing. The first release of the corpus contains over 2.8 million news articles and an additional 1 million short snippets (photo captions, video descriptions, etc.) published between 2001--2022 and collected from Voice of America's news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 13 of its 32 references that are in the map. First posted on arXiv 14 Jan 2022.

Builds on

13 of 32 references.

References · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  4. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  5. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  6. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  7. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  8. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  9. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  10. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  11. 2020KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundicited by 31
  12. 2020Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasetscited by 12
1 more
  1. 2021Manually Annotated Spelling Error Corpus for Amhariccited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 2.

10 papers.

  1. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  2. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  3. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  4. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  5. 2023MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languagescited by 23
  6. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  7. 2025SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?cited by 10
  8. 2022LR-Sum: Summarization for Less-Resourced Languagescited by 5
  9. 2022Extended Multilingual Protest News Detection -- Shared Task 1, CASE 2021 and 2022cited by 3
  10. 2025Comparing Approaches to Automatic Summarization in Less-Resourced Languagescited by 0