A Multilingual Evaluation of NER Robustness to Adversarial Inputs
Not on the map. Why?
Abstract (arXiv)
Adversarial evaluations of language models typically focus on English alone. In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its robustness to small perturbations in the input. Our results showed the NER models we explored across three languages (English, German and Hindi) are not very robust to such changes, as indicated by the fluctuations in the overall F1 score as well as in a more fine-grained evaluation. With that knowledge, we further explored whether it is possible to improve the existing NER models using a part of the generated adversarial data sets as augmented training data to train a new NER model or as fine-tuning data to adapt an existing NER model. Our results showed that both these approaches improve performance on the original as well as adversarial test sets. While there is no significant difference between the two approaches for English, re-training is significantly better than fine-tuning for German and Hindi.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 30 May 2023.
Builds on
25 of 43 references.
References · 25
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2023BloombergGPT: A Large Language Model for Financecited by 939
- 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
- 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
- 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
- 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
- 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
- 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
- 2017Generating Natural Adversarial Examplescited by 183
- 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
13 more
- 2017Deep Text Classification Can be Fooledcited by 106
- 2019On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Modelscited by 56
- 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
- 2022T-NER: An All-Round Python Library for Transformer-based Named Entity Recognitioncited by 37
- 2021RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Modelscited by 28
- 2017Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Taskcited by 24
- 2022A Survey of Adversarial Defences and Robustness in NLPcited by 23
- 2018Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehensioncited by 19
- 2020Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Modelscited by 13
- 2022What do we Really Know about State of the Art NER?cited by 10
- 2022Paraphrasing Techniques for Maritime QA systemcited by 7
- 2019Biomedical Named Entity Recognition via Reference-Set Augmented Bootstrappingcited by 4
- 2020A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restorationcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.