# Craw4LLM: Efficient Web Crawling for LLM Pretraining

arXiv: https://arxiv.org/abs/2502.13347 · HTML: https://mlsanity.com/paper/2502.13347/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [Axioms for Centrality](https://mlsanity.com/paper/1308.2140/) (2013)

## Built on by (most cited first, 6 of 6)

- [RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs](https://mlsanity.com/paper/2507.03253/) (2025)
- [TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training](https://mlsanity.com/paper/2508.17677/) (2025)
- [ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning](https://mlsanity.com/paper/2510.18250/) (2025)
- [Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality](https://mlsanity.com/paper/2606.11499/) (2026)
- [Document Quality Scoring for Web Crawling](https://mlsanity.com/paper/2504.11011/) (2025)
- [An LLM-Powered Agent for Real-Time Analysis of the Vietnamese IT Job Market](https://mlsanity.com/paper/2511.14767/) (2025)

## Abstract (arXiv)

Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper presents Craw4LLM, an efficient web crawling method that explores the web graph based on the preference of LLM pretraining. Specifically, it leverages the influence of a webpage in LLM pretraining as the priority score of the web crawler's scheduler, replacing the standard graph connectivity based priority. Our experiments on a web graph containing 900 million webpages from a commercial search engine's index demonstrate the efficiency of Craw4LLM in obtaining high-quality pretraining data. With just 21% URLs crawled, LLMs pretrained on Craw4LLM data reach the same downstream performances of previous crawls, significantly reducing the crawling waste and alleviating the burdens on websites. Our code is publicly available at https://github.com/cxcscmu/Craw4LLM.