# How to Evaluate Entity Resolution Systems: An Entity-Centric Framework with Application to Inventor Name Disambiguation

arXiv: https://arxiv.org/abs/2404.05622 · HTML: https://mlsanity.com/paper/2404.05622/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 67 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 11 of 11)

- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Automated Data Slicing for Model Validation:A Big data - AI Integration Approach](https://mlsanity.com/paper/1807.06068/) (2018)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components](https://mlsanity.com/paper/1908.01094/) (2019)
- [Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction](https://mlsanity.com/paper/2205.05889/) (2022)
- [Deepchecks: A Library for Testing and Validating Machine Learning Models and Data](https://mlsanity.com/paper/2203.08491/) (2022)
- [In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling](https://mlsanity.com/paper/1703.00617/) (2017)
- [Estimating the Performance of Entity Resolution Algorithms: Lessons Learned Through PatentsView.org](https://mlsanity.com/paper/2210.01230/) (2022)
- [PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation](https://mlsanity.com/paper/2301.03591/) (2023)

## Built on by (most cited first, 1 of 1)

- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)

## Abstract (arXiv)

Entity resolution (record linkage, microclustering) systems are notoriously difficult to evaluate. Looking for a needle in a haystack, traditional evaluation methods use sophisticated, application-specific sampling schemes to find matching pairs of records among an immense number of non-matches. We propose an alternative that facilitates the creation of representative, reusable benchmark data sets without necessitating complex sampling schemes. These benchmark data sets can then be used for model training and a variety of evaluation tasks. Specifically, we propose an entity-centric data labeling methodology that integrates with a unified framework for monitoring summary statistics, estimating key performance metrics such as cluster and pairwise precision and recall, and analyzing root causes for errors. We validate the framework in an application to inventor name disambiguation and through simulation studies. Software: https://github.com/OlivierBinette/er-evaluation/