# Toxicity of the Commons: Curating Open-Source Pre-Training Data arXiv: https://arxiv.org/abs/2410.22587 ยท HTML: https://mlsanity.com/paper/2410.22587/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 68 references that are in the map. - First posted on arXiv 29 Oct 2024. ## Builds on (most cited first, 30 of 32) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023) - [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023) - [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021) - [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021) - [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024) - [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024) - [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024) - [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021) - [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024) - [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024) - [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020) - [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024) - [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023) - [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022) - [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023) - [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021) - [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021) - [A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems](https://mlsanity.com/paper/2402.18649/) (2024) - [OpenELM: An Efficient Language Model Family with Open Training and Inference Framework](https://mlsanity.com/paper/2404.14619/) (2024) - [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024) - [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024) - [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023) - [BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B](https://mlsanity.com/paper/2311.00117/) (2023) - [Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification](https://mlsanity.com/paper/2011.06485/) (2020) - [The 2024 Foundation Model Transparency Index](https://mlsanity.com/paper/2407.12929/) (2024) - [Algorithmic Injustices: Towards a Relational Ethics](https://mlsanity.com/paper/1912.07376/) (2019) - [QueerBench: Quantifying Discrimination in Language Models Toward Queer Identities](https://mlsanity.com/paper/2406.12399/) (2024) ## Built on by (most cited first, 7 of 7) - [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025) - [Enhancing Multilingual LLM Pretraining with Model-Based Data Selection](https://mlsanity.com/paper/2502.10361/) (2025) - [Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training](https://mlsanity.com/paper/2506.01732/) (2025) - [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025) - [The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models](https://mlsanity.com/paper/2504.07854/) (2025) - [The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models](https://mlsanity.com/paper/2510.13996/) (2025) - [Training a General Purpose Automated Red Teaming Model](https://mlsanity.com/paper/2604.23067/) (2026) ## Abstract (arXiv) Open-source large language models are becoming increasingly available and popular among researchers and practitioners. While significant progress has been made on open-weight models, open training data is a practice yet to be adopted by the leading open-weight models creators. At the same time, there researchers are working to make language models safer. We propose a data curation pipeline to reduce harmful outputs by models trained on public domain data. There are unique challenges to working with public domain data, as these sources differ from web text in both form and content. Many sources are historical documents and are the result of Optical Character Recognition (OCR). Consequently, current state-of-the-art approaches to toxicity filtering are often infeasible or inappropriate for open data models. In this paper, we introduce a new fully open-source pipeline for open-data toxicity filtering. Our contributions are threefold. We create a custom training dataset, ToxicCommons, which is composed of texts which have been classified across five different dimensions (racial/origin-based, gender/sex-based, religious, ability-based discrimination, and violence). We use this dataset to train a custom classifier, Celadon, that can be used to detect toxic content in open data more efficiently at a larger scale. Finally, we describe the balanced approach to content filtration that optimizes safety filtering with respect to the filtered data available for training.