# Incremental Knowledge Base Construction Using DeepDive

arXiv: https://arxiv.org/abs/1502.00731 · HTML: https://mlsanity.com/paper/1502.00731/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2015.

## Built on by (most cited first, 16 of 16)

- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Learning Hyper Label Model for Programmatic Weak Supervision](https://mlsanity.com/paper/2207.13545/) (2022)
- [Ground Truth Inference for Weakly Supervised Entity Matching](https://mlsanity.com/paper/2211.06975/) (2022)
- [Interactive Machine Comprehension with Dynamic Knowledge Graphs](https://mlsanity.com/paper/2109.00077/) (2021)
- [DeeperDive: The Unreasonable Effectiveness of Weak Supervision in Document Understanding A Case Study in Collaboration with UiPath Inc](https://mlsanity.com/paper/2208.08000/) (2022)
- [Entity Context Graph: Learning Entity Representations fromSemi-Structured Textual Sources on the Web](https://mlsanity.com/paper/2103.15950/) (2021)
- [Reliable Programmatic Weak Supervision with Confidence Intervals for Label Probabilities](https://mlsanity.com/paper/2508.03896/) (2025)
- [Multi-Scale Feature and Metric Learning for Relation Extraction](https://mlsanity.com/paper/2107.13425/) (2021)
- [Common Sense Knowledge Learning for Open Vocabulary Neural Reasoning: A First View into Chronic Disease Literature](https://mlsanity.com/paper/2111.13781/) (2021)
- [Regularized Data Programming with Automated Bayesian Prior Selection](https://mlsanity.com/paper/2210.08677/) (2022)
- [RACH-Space: Reconstructing Adaptive Convex Hull Space with Applications in Weak Supervision](https://mlsanity.com/paper/2307.04870/) (2023)
- [Where Do Your Citations Come From? Citation-Constellation: A Free, Open-Source, No-Code, and Auditable Tool for Citation Network Decomposition with Complementary BARON and HEROCON Scores](https://mlsanity.com/paper/2603.24216/) (2026)
- [Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction](https://mlsanity.com/paper/2608.08459/) (2026)
- [RegExtract: A Benchmark for PFAS Regulatory Information Extraction with LLMs](https://mlsanity.com/paper/2610.15103/) (2026)
- [TL;QR: Querying Temporal Discursive Patterns in Textual Corpora](https://mlsanity.com/paper/2610.15712/) (2026)

## Abstract (arXiv)

Populating a database with unstructured information is a long-standing problem in industry and research that encompasses problems of extraction, cleaning, and integration. Recent names used for this problem include dealing with dark data and knowledge base construction (KBC). In this work, we describe DeepDive, a system that combines database and machine learning ideas to help develop KBC systems, and we present techniques to make the KBC process more efficient. We observe that the KBC process is iterative, and we develop techniques to incrementally produce inference results for KBC systems. We propose two methods for incremental inference, based respectively on sampling and variational techniques. We also study the tradeoff space of these methods and develop a simple rule-based optimizer. DeepDive includes all of these contributions, and we evaluate DeepDive on five KBC systems, showing that it can speed up KBC inference tasks by up to two orders of magnitude with negligible impact on quality.