# A Multilingual Evaluation of NER Robustness to Adversarial Inputs

arXiv: https://arxiv.org/abs/2305.18933 · HTML: https://mlsanity.com/paper/2305.18933/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://mlsanity.com/paper/1805.02266/) (2018)
- [Deep Text Classification Can be Fooled](https://mlsanity.com/paper/1704.08006/) (2017)
- [On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models](https://mlsanity.com/paper/1903.06620/) (2019)
- [A Challenge Set Approach to Evaluating Machine Translation](https://mlsanity.com/paper/1704.07431/) (2017)
- [T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition](https://mlsanity.com/paper/2209.12616/) (2022)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task](https://mlsanity.com/paper/1711.01505/) (2017)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension](https://mlsanity.com/paper/1808.08744/) (2018)
- [Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2004.04123/) (2020)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [Paraphrasing Techniques for Maritime QA system](https://mlsanity.com/paper/2203.10854/) (2022)
- [Biomedical Named Entity Recognition via Reference-Set Augmented Bootstrapping](https://mlsanity.com/paper/1906.00282/) (2019)
- [A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration](https://mlsanity.com/paper/2010.02864/) (2020)

## Built on by (most cited first, 2 of 2)

- [Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets](https://mlsanity.com/paper/2403.20056/) (2024)
- [Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution](https://mlsanity.com/paper/2503.03201/) (2025)

## Abstract (arXiv)

Adversarial evaluations of language models typically focus on English alone. In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its robustness to small perturbations in the input. Our results showed the NER models we explored across three languages (English, German and Hindi) are not very robust to such changes, as indicated by the fluctuations in the overall F1 score as well as in a more fine-grained evaluation. With that knowledge, we further explored whether it is possible to improve the existing NER models using a part of the generated adversarial data sets as augmented training data to train a new NER model or as fine-tuning data to adapt an existing NER model. Our results showed that both these approaches improve performance on the original as well as adversarial test sets. While there is no significant difference between the two approaches for English, re-training is significantly better than fine-tuning for German and Hindi.