# A Critical Re-evaluation of Benchmark Datasets for (Deep) Learning-Based Matching Algorithms

arXiv: https://arxiv.org/abs/2307.01231 · HTML: https://mlsanity.com/paper/2307.01231/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 68 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Deep Entity Matching with Pre-Trained Language Models](https://mlsanity.com/paper/2004.00584/) (2020)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [DeepER -- Deep Entity Resolution](https://mlsanity.com/paper/1710.00597/) (2017)
- [Recurrent Highway Networks](https://mlsanity.com/paper/1607.03474/) (2016)
- [Low-resource Deep Entity Resolution with Transfer and Active Learning](https://mlsanity.com/paper/1906.08042/) (2019)
- [Neural Networks for Entity Matching: A Survey](https://mlsanity.com/paper/2010.11075/) (2020)
- [A Benchmark and Comprehensive Survey on Knowledge Graph Entity Alignment via Representation Learning](https://mlsanity.com/paper/2103.15059/) (2021)
- [ZeroER: Entity Resolution using Zero Labeled Examples](https://mlsanity.com/paper/1908.06049/) (2019)
- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Modeling Voting for System Combination in Machine Translation](https://mlsanity.com/paper/2007.06943/) (2020)
- [Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction](https://mlsanity.com/paper/2205.05889/) (2022)
- [Deep Indexed Active Learning for Matching Heterogeneous Entity Representations](https://mlsanity.com/paper/2104.03986/) (2021)
- [CorDEL: A Contrastive Deep Learning Approach for Entity Linkage](https://mlsanity.com/paper/2009.07203/) (2020)
- [Interpretable and Low-Resource Entity Matching via Decoupling Feature Learning from Decision Making](https://mlsanity.com/paper/2106.04174/) (2021)
- [problexity -- an open-source Python library for binary classification problem complexity assessment](https://mlsanity.com/paper/2207.06709/) (2022)

## Built on by (most cited first, 4 of 4)

- [Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching](https://mlsanity.com/paper/2405.16884/) (2024)
- [WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Models](https://mlsanity.com/paper/2505.09595/) (2025)
- [ICE-ID: A Novel Historical Census Dataset for Longitudinal Identity Resolution](https://mlsanity.com/paper/2506.13792/) (2025)
- [Beyond Traditional Algorithms: Leveraging LLMs for Accurate Cross-Border Entity Identification](https://mlsanity.com/paper/2507.11086/) (2025)

## Abstract (arXiv)

Entity resolution (ER) is the process of identifying records that refer to the same entities within one or across multiple databases. Numerous techniques have been developed to tackle ER challenges over the years, with recent emphasis placed on machine and deep learning methods for the matching phase. However, the quality of the benchmark datasets typically used in the experimental evaluations of learning-based matching algorithms has not been examined in the literature. To cover this gap, we propose four different approaches to assessing the difficulty and appropriateness of 13 established datasets: two theoretical approaches, which involve new measures of linearity and existing measures of complexity, and two practical approaches: the difference between the best non-linear and linear matchers, as well as the difference between the best learning-based matcher and the perfect oracle. Our analysis demonstrates that most of the popular datasets pose rather easy classification tasks. As a result, they are not suitable for properly evaluating learning-based matching algorithms. To address this issue, we propose a new methodology for yielding benchmark datasets. We put it into practice by creating four new matching tasks, and we verify that these new benchmarks are more challenging and therefore more suitable for further advancements in the field.