# Probabilistic Record Linkage and Deduplication after Indexing, Blocking, and Filtering

arXiv: https://arxiv.org/abs/1603.07816 · HTML: https://mlsanity.com/paper/1603.07816/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2016.

## Built on by (most cited first, 4 of 4)

- [Multifile Partitioning for Record Linkage and Duplicate Detection](https://mlsanity.com/paper/2110.03839/) (2021)
- [A Primer on the Data Cleaning Pipeline](https://mlsanity.com/paper/2307.13219/) (2023)
- [Alternating minimization algorithm with initialization analysis for r-local and k-sparse unlabeled sensing](https://mlsanity.com/paper/2211.07621/) (2022)
- [Enhancing Equitable Access to AI in Housing and Homelessness System of Care through Federated Learning](https://mlsanity.com/paper/2408.07845/) (2024)

## Abstract (arXiv)

Probabilistic record linkage, the task of merging two or more databases in the absence of a unique identifier, is a perennial and challenging problem. It is closely related to the problem of deduplicating a single database, which can be cast as linking a single database against itself. In both cases the number of possible links grows rapidly in the size of the databases under consideration, and in most applications it is necessary to first reduce the number of record pairs that will be compared. Spurred by practical considerations, a range of methods have been developed for this task. These methods go under a variety of names, including indexing and blocking, and have seen significant development. However, methods for inferring linkage structure that account for indexing, blocking, and additional filtering steps have not seen commensurate development. In this paper we review the implications of indexing, blocking and filtering within the popular Fellegi-Sunter framework, and propose a new model to account for particular forms of indexing and filtering.