Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics
Not on the map. Why?
Abstract (arXiv)
PDF corpora advertise their size in tokens, but every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) is computed per document, and none decomposes its token total. Because PDF length is extremely skewed, the two units can describe the same corpus very differently. We ask how the headline statistics of a web-PDF corpus change when each document is weighted by the text it contributes rather than counted once. We used CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M Common Crawl PDFs, 32.6B tokens), the one public corpus that pairs the fragments Common Crawl stored with the re-fetched originals. Text mass is highly concentrated: 3.02% of text-bearing documents hold half the tokens (Gini 0.807). The clearest consequence is Common Crawl's payload cap, which truncated 23.06% of these documents but 63.08% of their text. Reconstructing the truncated fragments and extracting both versions, two widely used text-layer parsers recover only 1.4% and 11.4% of that exposed text, so roughly 55-62% of the corpus's text is unrecoverable from the crawl by such pipelines; under the 5MiB cap adopted in March 2025, 30.19% of tokens would still be exposed. We recommend that corpus statistics be reported in both units, documents and tokens.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 12 references that are in the map. First posted on arXiv 17 Aug 2026.
Builds on
5 of 12 references.
References · 5
- 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
- 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 150
- 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
- 2023CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Datacited by 21
- 2025GovScape: A Public Multimodal Search System for 70 Million Pages of Government PDFscited by 1
Built on by
Not cited yet.