# A Robust and Efficient Pipeline for Enterprise-Level Large-Scale Entity Resolution

arXiv: https://arxiv.org/abs/2508.03767 · HTML: https://mlsanity.com/paper/2508.03767/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 28 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 2 of 2)

- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Convex Hierarchical Clustering for Graph-Structured Data](https://mlsanity.com/paper/1911.03417/) (2019)

## Abstract (arXiv)

Entity resolution (ER) remains a significant challenge in data management, especially when dealing with large datasets. This paper introduces MERAI (Massive Entity Resolution using AI), a robust and efficient pipeline designed to address record deduplication and linkage issues in high-volume datasets at an enterprise level. The pipeline's resilience and accuracy have been validated through various large-scale record deduplication and linkage projects. To evaluate MERAI's performance, we compared it with two well-known entity resolution libraries, Dedupe and Splink. While Dedupe failed to scale beyond 2 million records due to memory constraints, MERAI successfully processed datasets of up to 15.7 million records and produced accurate results across all experiments. Experimental data demonstrates that MERAI outperforms both baseline systems in terms of matching accuracy, with consistently higher F1 scores in both deduplication and record linkage tasks. MERAI offers a scalable and reliable solution for enterprise-level large-scale entity resolution, ensuring data integrity and consistency in real-world applications.