# Interpretable Multi-dataset Evaluation for Named Entity Recognition

arXiv: https://arxiv.org/abs/2011.06854 · HTML: https://mlsanity.com/paper/2011.06854/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2020.

## Built on by (most cited first, 27 of 27)

- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [SpanNER: Named Entity Re-/Recognition as Span Prediction](https://mlsanity.com/paper/2106.00641/) (2021)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice](https://mlsanity.com/paper/2404.16958/) (2024)
- [Interpreting Language Models Through Knowledge Graph Extraction](https://mlsanity.com/paper/2111.08546/) (2021)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [KPI-EDGAR: A Novel Dataset and Accompanying Metric for Relation Extraction from Financial Documents](https://mlsanity.com/paper/2210.09163/) (2022)
- [DataLab: A Platform for Data Analysis and Intervention](https://mlsanity.com/paper/2202.12875/) (2022)
- [hmBERT: Historical Multilingual Language Models for Named Entity Recognition](https://mlsanity.com/paper/2205.15575/) (2022)
- [How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?](https://mlsanity.com/paper/2101.00160/) (2021)
- [Polyglot Prompt: Multilingual Multitask PrompTraining](https://mlsanity.com/paper/2204.14264/) (2022)
- [Regularization for Long Named Entity Recognition](https://mlsanity.com/paper/2104.07249/) (2021)
- [Larger-Context Tagging: When and Why Does It Work?](https://mlsanity.com/paper/2104.04434/) (2021)
- [Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction](https://mlsanity.com/paper/2109.12008/) (2021)
- [Win-Win Cooperation: Bundling Sequence and Span Models for Named Entity Recognition](https://mlsanity.com/paper/2207.03300/) (2022)
- [Statistical Dataset Evaluation: Reliability, Difficulty, and Validity](https://mlsanity.com/paper/2212.09272/) (2022)
- [Enhancing Label Consistency on Document-level Named Entity Recognition](https://mlsanity.com/paper/2210.12949/) (2022)
- [Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction](https://mlsanity.com/paper/2308.02951/) (2023)
- [BTPK-based interpretable method for NER tasks based on Talmudic Public Announcement Logic](https://mlsanity.com/paper/2201.09523/) (2022)
- [Multi-Dimensional Ability Diagnosis for Machine Learning Algorithms](https://mlsanity.com/paper/2307.07134/) (2023)
- [Named Clinical Entity Recognition Benchmark](https://mlsanity.com/paper/2410.05046/) (2024)
- [Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks](https://mlsanity.com/paper/2602.12759/) (2026)
- [On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence](https://mlsanity.com/paper/2610.00007/) (2026)

## Abstract (arXiv)

With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform differently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them, identifying the strengths and weaknesses of current systems. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area: https://github.com/neulab/InterpretEval.