mlsanityData to 14 Oct 2026

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

arXiv 2303.03915 ↗First posted 7 Mar 2023cs.CL cs.AIcited by 148

Not on the map. Why?

Abstract (arXiv)

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

Key facts. Cited by 148 papers in the map (data through 14 Oct 2026). Builds on 59 of its 136 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

59 of 136 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2016Bag of Tricks for Efficient Text Classificationcited by 876
38 more
  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  3. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  7. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  9. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  13. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  14. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  15. 2021The Values Encoded in Machine Learning Researchcited by 172
  16. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  17. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  18. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  19. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  20. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  21. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 126
  22. 2020Participation is not a Design Fix for Machine Learningcited by 111
  23. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  24. 2019On the Use of ArXiv as a Datasetcited by 103
  25. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  26. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 75
  27. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  28. 2018Universal Language Model Fine-tuning for Text Classificationcited by 52
  29. 2021Changing the World by Changing the Datacited by 51
  30. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 47
  31. 2020Behavioral Use Licensing for Responsible AIcited by 46
  32. 2020AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languagescited by 44
  33. 2022Datasheet for the Pilecited by 43
  34. 20161.5 billion words Arabic Corpuscited by 35
  35. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 33
  36. 2021Envisioning Communities: A Participatory Approach Towards AI for Social Goodcited by 30
  37. 2022BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Samplingcited by 30
  38. 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 55; 2024: 47; 2025: 29; 2026: 17.

148 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2023Scaling Data-Constrained Language Modelscited by 403
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  8. 2024What matters when building vision-language models?cited by 326
  9. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  10. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 287
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
38 more
  1. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  2. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  3. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  4. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  5. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  6. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  7. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  8. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  9. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  10. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  11. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  12. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 75
  13. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 67
  14. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  15. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  16. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  17. 2023nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Modelscited by 60
  18. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  19. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  20. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  21. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  22. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  23. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  24. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 51
  25. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 49
  26. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
  27. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 46
  28. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  29. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  30. 2024Proving membership in LLM pretraining data via data watermarkscited by 40
  31. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  32. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  33. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
  34. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  35. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  36. 2023CBQ: Cross-Block Quantization for Large Language Modelscited by 34
  37. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  38. 2023QuantEase: Optimization-based Quantization for Language Modelscited by 29

50 most cited of 148 All 148 (JSON)