# Deduplication in a massive clinical note dataset

arXiv: https://arxiv.org/abs/1704.05617 · HTML: https://mlsanity.com/paper/1704.05617/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2017.

## Built on by (most cited first, 1 of 1)

- [Noise-Robust De-Duplication at Scale](https://mlsanity.com/paper/2210.04261/) (2022)

## Abstract (arXiv)

Duplication, whether exact or partial, is a common issue in many datasets. In clinical notes data, duplication (and near duplication) can arise for many reasons, such as the pervasive use of templates, copy-pasting, or notes being generated by automated procedures. A key challenge in removing such near duplicates is the size of such datasets; our own dataset consists of more than 10 million notes. To detect and correct such duplicates requires algorithms that both accurate and highly scalable. We describe a solution based on Minhashing with Locality Sensitive Hashing. In this paper, we present the theory behind this method and present a database-inspired approach to make the method scalable. We also present a clustering technique using disjoint sets to produce dense clusters, which speeds up our algorithm.