mlsanityData to 7 Oct 2026

A New Massive Multilingual Dataset for High-Performance Language Technologies

arXiv 2403.14009 ↗First posted 20 Mar 2024cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ~5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 12 of its 35 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

12 of 35 references.

References · 12

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  7. 2018Marian: Fast Neural Machine Translation in C++cited by 202
  8. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  9. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  10. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  11. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  12. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 21; 2026: 24.

56 papers.

Sort:

By area:

  1. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  2. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 28
  3. 2024Meltemi: The first open Large Language Model for Greekcited by 16
  4. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  5. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 15
  6. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  7. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  8. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  9. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  10. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  11. 2025Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basquecited by 12
  12. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
38 more
  1. 2024The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspectivecited by 8
  2. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 7
  3. 2025DocHPLT: A Massively Multilingual Document-Level Translation Datasetcited by 7
  4. 2026EuroLLM-22B: Technical Reportcited by 7
  5. 2024A Survey of Large Language Models for European Languagescited by 6
  6. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 6
  7. 2025DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detectioncited by 5
  8. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 5
  9. 2025Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scalecited by 5
  10. 2025Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probabilitycited by 4
  11. 2025ALGEN: Few-shot Inversion Attacks on Textual Embeddings using Alignment and Generationcited by 4
  12. 2025Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Reviewcited by 4
  13. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 4
  14. 2026Zero2Text: Zero-Training Cross-Domain Inversion Attacks on Textual Embeddingscited by 4
  15. 2026OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Reportcited by 4
  16. 2025skLEP: A Slovak General Language Understanding Benchmarkcited by 3
  17. 2025From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Taskcited by 3
  18. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  19. 2024FastSpell: the LangId Magic Spellcited by 1
  20. 2024SWEb: A Large Web Dataset for the Scandinavian Languagescited by 1
  21. 2025Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?cited by 1
  22. 2025ACADATA: Parallel Dataset of Academic Data for Machine Translationcited by 1
  23. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 1
  24. 2026GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-trainingcited by 1
  25. 2026Polite on the Surface, Broken in Practice: A Curated Dataset for Fixing Generation and Register Failures in Low-Resource Bangla Text Generationcited by 1
  26. 2026DunbaaBERT: From Sacrifice to Semanticscited by 1
  27. 2024Data Processing for the OpenGPT-X Model Familycited by 0
  28. 2025BERnaT: Basque Encoders for Representing Natural Textual Diversitycited by 0
  29. 2026Sigmoid Head for Quality Estimation under Language Ambiguitycited by 0
  30. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  31. 2026The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corporacited by 0
  32. 2026AfroScope: A Framework for Studying the Linguistic Landscape of Africacited by 0
  33. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0
  34. 2026MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptationcited by 0
  35. 2026A Diversity Diet for a Healthier Model: A Case Study of French ModernBERTcited by 0
  36. 2026PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Developmentcited by 0
  37. 2026On the limited utility of parallel data for learning shared multilingual representationscited by 0
  38. 2026Open Machine Translation for Esperantocited by 0

50 most cited of 56 All 56 (JSON)