# Multifile Partitioning for Record Linkage and Duplicate Detection

arXiv: https://arxiv.org/abs/2110.03839 · HTML: https://mlsanity.com/paper/2110.03839/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Bayesian cluster analysis: Point estimation and credible balls](https://mlsanity.com/paper/1505.03339/) (2015)
- [Flexible Models for Microclustering with Application to Entity Resolution](https://mlsanity.com/paper/1610.09780/) (2016)
- [A Comparison of Blocking Methods for Record Linkage](https://mlsanity.com/paper/1407.3191/) (2014)
- [Random Partition Models for Microclustering Tasks](https://mlsanity.com/paper/2004.02008/) (2020)
- [Bayesian factorizations of big sparse tensors](https://mlsanity.com/paper/1306.1598/) (2013)
- [Bayesian Estimation of Bipartite Matchings for Record Linkage](https://mlsanity.com/paper/1601.06630/) (2016)
- [Probabilistic Record Linkage and Deduplication after Indexing, Blocking, and Filtering](https://mlsanity.com/paper/1603.07816/) (2016)
- [Scaling Bayesian Probabilistic Record Linkage with Post-Hoc Blocking: An Application to the California Great Registers](https://mlsanity.com/paper/1905.05337/) (2019)
- [Detecting duplicates in a homicide registry using a Bayesian partitioning approach](https://mlsanity.com/paper/1407.8219/) (2014)
- [A Prior for Record Linkage Based on Allelic Partitions](https://mlsanity.com/paper/2008.10118/) (2020)
- [Microclustering: When the Cluster Sizes Grow Sublinearly with the Size of the Data Set](https://mlsanity.com/paper/1512.00792/) (2015)
- [d-blink: Distributed End-to-End Bayesian Entity Resolution](https://mlsanity.com/paper/1909.06039/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Primer on the Data Cleaning Pipeline](https://mlsanity.com/paper/2307.13219/) (2023)

## Abstract (arXiv)

Merging datafiles containing information on overlapping sets of entities is a challenging task in the absence of unique identifiers, and is further complicated when some entities are duplicated in the datafiles. Most approaches to this problem have focused on linking two files assumed to be free of duplicates, or on detecting which records in a single file are duplicates. However, it is common in practice to encounter scenarios that fit somewhere in between or beyond these two settings. We propose a Bayesian approach for the general setting of multifile record linkage and duplicate detection. We use a novel partition representation to propose a structured prior for partitions that can incorporate prior information about the data collection processes of the datafiles in a flexible manner, and extend previous models for comparison data to accommodate the multifile setting. We also introduce a family of loss functions to derive Bayes estimates of partitions that allow uncertain portions of the partitions to be left unresolved. The performance of our proposed methodology is explored through extensive simulations. Code implementing the methodology is available at https://github.com/aleshing/multilink .