# Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models

arXiv: https://arxiv.org/abs/2004.04123 · HTML: https://mlsanity.com/paper/2004.04123/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Neural Data Augmentation via Example Extrapolation](https://mlsanity.com/paper/2102.01335/) (2021)
- [T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition](https://mlsanity.com/paper/2209.12616/) (2022)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition](https://mlsanity.com/paper/2107.11610/) (2021)
- [On the Robustness of Reading Comprehension Models to Entity Renaming](https://mlsanity.com/paper/2110.08555/) (2021)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?](https://mlsanity.com/paper/2101.00160/) (2021)
- [Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?](https://mlsanity.com/paper/2305.12096/) (2023)
- [A Multilingual Evaluation of NER Robustness to Adversarial Inputs](https://mlsanity.com/paper/2305.18933/) (2023)
- [Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks](https://mlsanity.com/paper/2602.12759/) (2026)
- [Mind the Gap: Robustness Risks in PII Detection Systems](https://mlsanity.com/paper/2609.03464/) (2026)

## Abstract (arXiv)

Named entity recognition systems perform well on standard datasets comprising English news. But given the paucity of data, it is difficult to draw conclusions about the robustness of systems with respect to recognizing a diverse set of entities. We propose a method for auditing the in-domain robustness of systems, focusing specifically on differences in performance due to the national origin of entities. We create entity-switched datasets, in which named entities in the original texts are replaced by plausible named entities of the same type but of different national origin. We find that state-of-the-art systems' performance vary widely even in-domain: In the same context, entities from certain origins are more reliably recognized than entities from elsewhere. Systems perform best on American and Indian entities, and worst on Vietnamese and Indonesian entities. This auditing approach can facilitate the development of more robust named entity recognition systems, and will allow research in this area to consider fairness criteria that have received heightened attention in other predictive technology work.