mlsanityData to 7 Oct 2026

The ROOTS Search Tool: Data Transparency for LLMs

arXiv 2302.14035 ↗First posted 27 Feb 2023cs.CL cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

ROOTS is a 1.6TB multilingual text corpus developed for the training of BLOOM, currently the largest language model explicitly accompanied by commensurate data governance efforts. In continuation of these efforts, we present the ROOTS Search Tool: a search engine over the entire ROOTS corpus offering both fuzzy and exact search capabilities. ROOTS is the largest corpus to date that can be investigated this way. The ROOTS Search Tool is open-sourced and available on Hugging Face Spaces. We describe our implementation and the possible use cases of our tool.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

35 of 59 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2018Datasheets for Datasetscited by 1,371
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
23 more
  1. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  7. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  9. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  10. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 176
  11. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  12. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  13. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  14. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  15. 2021A Survey of Race, Racism, and Anti-Racism in NLPcited by 74
  16. 2021Changing the World by Changing the Datacited by 51
  17. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  18. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 32
  19. 2022Measuring Datacited by 18
  20. 2020Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Datasetcited by 15
  21. 2022BigScience: A Case Study in the Social Construction of a Multilingual Large Language Modelcited by 11
  22. 2022Quote Erat Demonstrandum: A Web Interface for Exploring the Quotebank Corpuscited by 5
  23. 2022esCorpius: A Massive Spanish Crawling Corpuscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 10; 2025: 5; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023StarCoder: may the source be with you!cited by 790
  2. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  5. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  6. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  7. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  8. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  9. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  10. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  11. 2023Data Portraits: Recording Foundation Model Training Datacited by 29
  12. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
18 more
  1. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  2. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  3. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  4. 2024A Taxonomy for Data Contamination in Large Language Modelscited by 11
  5. 2024Private Synthetic Text Generation with Diffusion Modelscited by 10
  6. 2025Data Leakage in Visual Datasetscited by 7
  7. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  8. 2023Cultural Adaptation of Recipescited by 6
  9. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluationcited by 4
  10. 2023Oasis: Data Curation and Assessment System for Pretraining of Large Language Modelscited by 3
  11. 2024Towards Transparency: Exploring LLM Trainings Datasets through Visual Topic Modeling and Semantic Framecited by 1
  12. 2024What's New in My Data? Novelty Exploration via Contrastive Generationcited by 1
  13. 2025Beyond Release: Access Considerations for Generative AI Systemscited by 1
  14. 2026TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMscited by 1
  15. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  16. 2025Getting Your Indices in a Row: Full-Text Search for LLM Training Data for Real Worldcited by 0
  17. 2026Personal Information Parroting in Language Modelscited by 0
  18. 2026Auditing Chinese Web-scale Corpora via Sampled BPE Token Statisticscited by 0