# Host-Aware Web Crawling for LLM Pretraining Data

arXiv: https://arxiv.org/abs/2610.13321 · HTML: https://mlsanity.com/paper/2610.13321/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection](https://mlsanity.com/paper/2201.10474/) (2022)
- [Enhancing Multilingual LLM Pretraining with Model-Based Data Selection](https://mlsanity.com/paper/2502.10361/) (2025)
- [AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters](https://mlsanity.com/paper/2401.06408/) (2024)
- [ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information](https://mlsanity.com/paper/2211.15848/) (2022)
- [Axioms for Centrality](https://mlsanity.com/paper/1308.2140/) (2013)
- [Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models](https://mlsanity.com/paper/2505.22232/) (2025)
- [Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality](https://mlsanity.com/paper/2606.11499/) (2026)
- [Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls](https://mlsanity.com/paper/2607.13636/) (2026)

## Abstract (arXiv)

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.