# Extensive Error Analysis and a Learning-Based Evaluation of Medical Entity Recognition Systems to Approximate User Experience

arXiv: https://arxiv.org/abs/2006.05281 · HTML: https://mlsanity.com/paper/2006.05281/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 2 of 2)

- [A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning](https://mlsanity.com/paper/2105.09198/) (2021)
- [Stress Testing BERT Anaphora Resolution Models for Reaction Extraction in Chemical Patents](https://mlsanity.com/paper/2306.13379/) (2023)

## Abstract (arXiv)

When comparing entities extracted by a medical entity recognition system with gold standard annotations over a test set, two types of mismatches might occur, label mismatch or span mismatch. Here we focus on span mismatch and show that its severity can vary from a serious error to a fully acceptable entity extraction due to the subjectivity of span annotations. For a domain-specific BERT-based NER system, we showed that 25% of the errors have the same labels and overlapping span with gold standard entities. We collected expert judgement which shows more than 90% of these mismatches are accepted or partially accepted by the user. Using the training set of the NER system, we built a fast and lightweight entity classifier to approximate the user experience of such mismatches through accepting or rejecting them. The decisions made by this classifier are used to calculate a learning-based F-score which is shown to be a better approximation of a forgiving user's experience than the relaxed F-score. We demonstrated the results of applying the proposed evaluation metric for a variety of deep learning medical entity recognition models trained with two datasets.