# Corpus Conversion Service: A Machine Learning Platform to Ingest Documents at Scale

arXiv: https://arxiv.org/abs/1806.02284 · HTML: https://mlsanity.com/paper/1806.02284/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 12 of 12)

- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [TableFormer: Table Structure Understanding with Transformers](https://mlsanity.com/paper/2203.01017/) (2022)
- [OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition](https://mlsanity.com/paper/2403.19128/) (2024)
- [Optimized Table Tokenization for Table Structure Recognition](https://mlsanity.com/paper/2305.03393/) (2023)
- [VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups](https://mlsanity.com/paper/2106.00676/) (2021)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [Robust PDF Document Conversion Using Recurrent Neural Networks](https://mlsanity.com/paper/2102.09395/) (2021)
- [Delivering Document Conversion as a Cloud Service with High Throughput and Responsiveness](https://mlsanity.com/paper/2206.00785/) (2022)
- [CoVA: Context-aware Visual Attention for Webpage Information Extraction](https://mlsanity.com/paper/2110.12320/) (2021)
- [ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents](https://mlsanity.com/paper/2305.14962/) (2023)
- [Understanding in Artificial Intelligence](https://mlsanity.com/paper/2101.06573/) (2021)
- [Extracting ontology-compliant knowledge from scientific text describing irradiated materials using large language models](https://mlsanity.com/paper/2609.17291/) (2026)

## Abstract (arXiv)

Over the past few decades, the amount of scientific articles and technical literature has increased exponentially in size. Consequently, there is a great need for systems that can ingest these documents at scale and make the contained knowledge discoverable. Unfortunately, both the format of these documents (e.g. the PDF format or bitmap images) as well as the presentation of the data (e.g. complex tables) make the extraction of qualitative and quantitive data extremely challenging. In this paper, we present a modular, cloud-based platform to ingest documents at scale. This platform, called the Corpus Conversion Service (CCS), implements a pipeline which allows users to parse and annotate documents (i.e. collect ground-truth), train machine-learning classification algorithms and ultimately convert any type of PDF or bitmap-documents to a structured content representation format. We will show that each of the modules is scalable due to an asynchronous microservice architecture and can therefore handle massive amounts of documents. Furthermore, we will show that our capability to gather ground-truth is accelerated by machine-learning algorithms by at least one order of magnitude. This allows us to both gather large amounts of ground-truth in very little time and obtain very good precision/recall metrics in the range of 99% with regard to content conversion to structured output. The CCS platform is currently deployed on IBM internal infrastructure and serving more than 250 active users for knowledge-engineering project engagements.