# A Primer on the Data Cleaning Pipeline

arXiv: https://arxiv.org/abs/2307.13219 · HTML: https://mlsanity.com/paper/2307.13219/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 102 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [DeepER -- Deep Entity Resolution](https://mlsanity.com/paper/1710.00597/) (2017)
- [Low-resource Deep Entity Resolution with Transfer and Active Learning](https://mlsanity.com/paper/1906.08042/) (2019)
- [Neural Networks for Entity Matching: A Survey](https://mlsanity.com/paper/2010.11075/) (2020)
- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [S2AND: A Benchmark and Evaluation System for Author Name Disambiguation](https://mlsanity.com/paper/2103.07534/) (2021)
- [Flexible Models for Microclustering with Application to Entity Resolution](https://mlsanity.com/paper/1610.09780/) (2016)
- [End-to-End Entity Resolution for Big Data: A Survey](https://mlsanity.com/paper/1905.06397/) (2019)
- [A Comparison of Blocking Methods for Record Linkage](https://mlsanity.com/paper/1407.3191/) (2014)
- [Ethnicity sensitive author disambiguation using semi-supervised learning](https://mlsanity.com/paper/1508.07744/) (2015)
- [Random Partition Models for Microclustering Tasks](https://mlsanity.com/paper/2004.02008/) (2020)
- [Bayesian Estimation of Bipartite Matchings for Record Linkage](https://mlsanity.com/paper/1601.06630/) (2016)
- [Probabilistic Record Linkage and Deduplication after Indexing, Blocking, and Filtering](https://mlsanity.com/paper/1603.07816/) (2016)
- [Scaling Bayesian Probabilistic Record Linkage with Post-Hoc Blocking: An Application to the California Great Registers](https://mlsanity.com/paper/1905.05337/) (2019)
- [Detecting duplicates in a homicide registry using a Bayesian partitioning approach](https://mlsanity.com/paper/1407.8219/) (2014)
- [A Prior for Record Linkage Based on Allelic Partitions](https://mlsanity.com/paper/2008.10118/) (2020)
- [Generalized Bayesian Record Linkage and Regression with Exact Error Propagation](https://mlsanity.com/paper/1810.04808/) (2018)
- [d-blink: Distributed End-to-End Bayesian Entity Resolution](https://mlsanity.com/paper/1909.06039/) (2019)
- [Blocking Methods Applied to Casualty Records from the Syrian Conflict](https://mlsanity.com/paper/1510.07714/) (2015)
- [Performance Bounds for Graphical Record Linkage](https://mlsanity.com/paper/1703.02679/) (2017)
- [Bayesian Propagation of Record Linkage Uncertainty into Population Size Estimation of Human Rights Violations](https://mlsanity.com/paper/1812.09590/) (2018)
- [A Bayesian Multi-Layered Record Linkage Procedure to Analyze Functional Status of Medicare Patients with Traumatic Brain Injury](https://mlsanity.com/paper/2005.08549/) (2020)
- [Multifile Partitioning for Record Linkage and Duplicate Detection](https://mlsanity.com/paper/2110.03839/) (2021)
- [On the Reliability of Multiple Systems Estimation for the Quantification of Modern Slavery](https://mlsanity.com/paper/2112.01594/) (2021)

## Built on by (most cited first, 1 of 1)

- [Leveraging MIMIC Datasets for Better Digital Health: A Review on Open Problems, Progress Highlights, and Future Promises](https://mlsanity.com/paper/2506.12808/) (2025)

## Abstract (arXiv)

The availability of both structured and unstructured databases, such as electronic health data, social media data, patent data, and surveys that are often updated in real time, among others, has grown rapidly over the past decade. With this expansion, the statistical and methodological questions around data integration, or rather merging multiple data sources, has also grown. Specifically, the science of the ``data cleaning pipeline'' contains four stages that allow an analyst to perform downstream tasks, predictive analyses, or statistical analyses on ``cleaned data.'' This article provides a review of this emerging field, introducing technical terminology and commonly used methods.