mlsanityData to 14 Oct 2026

An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)

arXiv 2503.10267 ↗First posted 13 Mar 2025cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

36 of 64 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
24 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  4. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  5. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  6. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  7. 2018Marian: Fast Neural Machine Translation in C++cited by 203
  8. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  9. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  10. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  11. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  12. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  13. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  14. 2023A Survey on Detection of LLMs-Generated Contentcited by 71
  15. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  16. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  17. 2023An Open Dataset and Model for Language Identificationcited by 50
  18. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  19. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  20. 2021On the Multilingual Capabilities of Very Large-Scale English Language Modelscited by 22
  21. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  22. 2023NorQuAD: Norwegian Question Answering Datasetcited by 7
  23. 2024Automatic register identification for the open web using multilingual deep learningcited by 4
  24. 2025A Collection of Question Answering Datasets for Norwegiancited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  2. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 8
  3. 2025DocHPLT: A Massively Multilingual Document-Level Translation Datasetcited by 7
  4. 2025Model-Aware Tokenizer Transfercited by 6
  5. 2025NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMscited by 4
  6. 2025MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sourcescited by 3
  7. 2025Less Is More: Reducing Token Counts Without Compromising Performancecited by 2
  8. 2025Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasetscited by 1
  9. 2026SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmarkcited by 1
  10. 2025Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Languagecited by 0
  11. 2025FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Modelscited by 0
  12. 2026BYOL: Bring Your Own Language Into LLMscited by 0
4 more
  1. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  2. 2026Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparitiescited by 0
  3. 2026Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selectioncited by 0
  4. 2026MixLID: Token-Level Language Identification in Mixed-Language Textcited by 0