# Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs arXiv: https://arxiv.org/abs/2505.02009 ยท HTML: https://mlsanity.com/paper/2505.02009/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. - Builds on 24 of its 31 references that are in the map. - First posted on arXiv 4 May 2025. ## Placed in - LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/) ## Builds on (most cited first, 24 of 24) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020) - [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023) - [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024) - [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021) - [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021) - [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022) - [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020) - [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023) - [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021) - [Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data](https://mlsanity.com/paper/2212.10440/) (2022) - [Data, Data Everywhere: A Guide for Pretraining Dataset Construction](https://mlsanity.com/paper/2407.06380/) (2024) - [It's All in the Embedding! Fake News Detection Using Document Embeddings](https://mlsanity.com/paper/2304.07781/) (2023) - [Suicidal Ideation Detection on Social Media: A Review of Machine Learning Methods](https://mlsanity.com/paper/2201.10515/) (2022) - [Illicit Darkweb Classification via Natural-language Processing: Classifying Illicit Content of Webpages based on Textual Information](https://mlsanity.com/paper/2312.04944/) (2023) ## Built on by (most cited first, 9 of 9) - [An Embarrassingly Simple Defense Against LLM Abliteration Attacks](https://mlsanity.com/paper/2505.19056/) (2025) - [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025) - [Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection](https://mlsanity.com/paper/2508.20766/) (2025) - [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026) - [Going over Fine Web with a Fine-Tooth Comb: Technical Report of Indexing Fine Web for Problematic Content Search and Retrieval](https://mlsanity.com/paper/2508.21788/) (2025) - [On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs](https://mlsanity.com/paper/2510.05864/) (2025) - [Safety Is Not Universal: The Selective Safety Trap in LLM Alignment](https://mlsanity.com/paper/2601.04389/) (2026) - [Phoenix-VL 1.5 Medium Technical Report](https://mlsanity.com/paper/2605.10391/) (2026) - [Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails](https://mlsanity.com/paper/2606.05936/) (2026) ## Abstract (arXiv) Large language models (LLMs) have become integral to various real-world applications, leveraging massive, web-sourced datasets like Common Crawl, C4, and FineWeb for pretraining. While these datasets provide linguistic data essential for high-quality natural language generation, they often contain harmful content, such as hate speech, misinformation, and biased narratives. Training LLMs on such unfiltered data risks perpetuating toxic behaviors, spreading misinformation, and amplifying societal biases which can undermine trust in LLM-driven applications and raise ethical concerns about their use. This paper presents a large-scale analysis of inappropriate content across these datasets, offering a comprehensive taxonomy that categorizes harmful webpages into Topical and Toxic based on their intent. We also introduce a prompt evaluation dataset, a high-accuracy Topical and Toxic Prompt (TTP), and a transformer-based model (HarmFormer) for harmful content filtering. Additionally, we create a new multi-harm open-ended toxicity benchmark (HAVOC) and provide crucial insights into how models respond to adversarial toxic inputs. We share TTP, TTP-Eval, HAVOC and a sample of C4 inferenced on HarmFormer. Our work offers insights into ensuring safer LLM pretraining and serves as a resource for Responsible AI (RAI) compliance.