mlsanityData to 7 Oct 2026

CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data

arXiv 2304.14953 ↗First posted 28 Apr 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

In recent years, the field of document understanding has progressed a lot. A significant part of this progress has been possible thanks to the use of language models pretrained on large amounts of documents. However, pretraining corpora used in the domain of document understanding are single domain, monolingual, or nonpublic. Our goal in this paper is to propose an efficient pipeline for creating a big-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl, as PDF files are the most canonical types of documents as considered in document understanding. We analysed extensively all of the steps of the pipeline and proposed a solution which is a trade-off between data quality and processing time. We also share a CCpdf corpus in a form or an index of PDF files along with a script for downloading them, which produces a collection useful for language model pretraining. The dataset and tools published with this paper offer researchers the opportunity to develop even better multilingual language models.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 28 Apr 2023.

Builds on

24 of 38 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  9. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  12. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
12 more
  1. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  2. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  3. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 152
  4. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  5. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  6. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  7. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  8. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  9. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  10. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53
  11. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  12. 2021CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Trainingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 10; 2026: 6.

21 papers.

Sort:

By area:

  1. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  2. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  3. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  4. 2024Docling Technical Reportcited by 29
  5. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  6. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  7. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  8. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  9. 2023Beyond Document Page Classification: Design, Datasets, and Challengescited by 10
  10. 2025CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognitioncited by 9
  11. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  12. 2024μgat: Improving Single-Page Document Parsing by Providing Multi-Page Contextcited by 7
9 more
  1. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generationcited by 5
  2. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
  3. 2026CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligencecited by 5
  4. 2025CommonForms: A Large, Diverse Dataset for Form Field Detectioncited by 3
  5. 2025Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMMcited by 2
  6. 2026ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formatscited by 2
  7. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  8. 2026MORE: A Multilingual Document Parsing Benchmark and Evaluationcited by 2
  9. 2026Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statisticscited by 0