# The ROOTS Search Tool: Data Transparency for LLMs

arXiv: https://arxiv.org/abs/2302.14035 · HTML: https://mlsanity.com/paper/2302.14035/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 59 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus](https://mlsanity.com/paper/2105.05241/) (2021)
- [Data Governance in the Age of Large-Scale Data-Driven Language Technology](https://mlsanity.com/paper/2206.03216/) (2022)

## Built on by (most cited first, 30 of 31)

- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Dated Data: Tracing Knowledge Cutoffs in Large Language Models](https://mlsanity.com/paper/2403.12958/) (2024)
- [Data Portraits: Recording Foundation Model Training Data](https://mlsanity.com/paper/2303.03919/) (2023)
- [Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation](https://mlsanity.com/paper/2406.14644/) (2024)
- [Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions](https://mlsanity.com/paper/2410.18966/) (2024)
- [Position: Key Claims in LLM Research Have a Long Tail of Footnotes](https://mlsanity.com/paper/2308.07120/) (2023)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [A Taxonomy for Data Contamination in Large Language Models](https://mlsanity.com/paper/2407.08716/) (2024)
- [Private Synthetic Text Generation with Diffusion Models](https://mlsanity.com/paper/2410.22971/) (2024)
- [Cultural Adaptation of Recipes](https://mlsanity.com/paper/2310.17353/) (2023)
- [Data Leakage in Visual Datasets](https://mlsanity.com/paper/2508.17416/) (2025)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [Establishing Trustworthiness: Rethinking Tasks and Model Evaluation](https://mlsanity.com/paper/2310.05442/) (2023)
- [Oasis: Data Curation and Assessment System for Pretraining of Large Language Models](https://mlsanity.com/paper/2311.12537/) (2023)
- [Towards Transparency: Exploring LLM Trainings Datasets through Visual Topic Modeling and Semantic Frame](https://mlsanity.com/paper/2406.06574/) (2024)
- [What's New in My Data? Novelty Exploration via Contrastive Generation](https://mlsanity.com/paper/2410.14765/) (2024)
- [Beyond Release: Access Considerations for Generative AI Systems](https://mlsanity.com/paper/2502.16701/) (2025)
- [TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs](https://mlsanity.com/paper/2605.24079/) (2026)
- [A Conceptual Framework for AI Capability Evaluations](https://mlsanity.com/paper/2506.18213/) (2025)
- [Getting Your Indices in a Row: Full-Text Search for LLM Training Data for Real World](https://mlsanity.com/paper/2510.09471/) (2025)
- [Personal Information Parroting in Language Models](https://mlsanity.com/paper/2602.20580/) (2026)

## Abstract (arXiv)

ROOTS is a 1.6TB multilingual text corpus developed for the training of BLOOM, currently the largest language model explicitly accompanied by commensurate data governance efforts. In continuation of these efforts, we present the ROOTS Search Tool: a search engine over the entire ROOTS corpus offering both fuzzy and exact search capabilities. ROOTS is the largest corpus to date that can be investigated this way. The ROOTS Search Tool is open-sourced and available on Hugging Face Spaces. We describe our implementation and the possible use cases of our tool.