# Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

arXiv: https://arxiv.org/abs/2107.11610 · HTML: https://mlsanity.com/paper/2107.11610/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 101 references that are in the map.
- First posted on arXiv 24 Jul 2021.

## Builds on (most cited first, 30 of 62)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)

## Built on by (most cited first, 19 of 19)

- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [End-to-End Self-Debiasing Framework for Robust NLU Training](https://mlsanity.com/paper/2109.02071/) (2021)
- [RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation](https://mlsanity.com/paper/2109.10164/) (2021)
- [Robustness of Demonstration-based Learning Under Limited Data Scenario](https://mlsanity.com/paper/2210.10693/) (2022)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [DUMB: A Benchmark for Smart Evaluation of Dutch Models](https://mlsanity.com/paper/2305.13026/) (2023)
- [How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding](https://mlsanity.com/paper/2109.05696/) (2021)
- [Can Large Language Models Really Recognize Your Name?](https://mlsanity.com/paper/2505.14549/) (2025)
- [BEADs: Bias Evaluation Across Domains](https://mlsanity.com/paper/2406.04220/) (2024)
- [NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation](https://mlsanity.com/paper/2111.05196/) (2021)
- [NERDA-Con: Extending NER models for Continual Learning -- Integrating Distinct Tasks and Updating Distribution Shifts](https://mlsanity.com/paper/2206.14607/) (2022)
- [Do "English" Named Entity Recognizers Work Well on Global Englishes?](https://mlsanity.com/paper/2404.13465/) (2024)
- [Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning](https://mlsanity.com/paper/2408.03819/) (2024)
- [Knowledge Distillation with Noisy Labels for Natural Language Understanding](https://mlsanity.com/paper/2109.10147/) (2021)
- [#maskUp: Selective Attribute Encryption for Sensitive Vocalization for English language on Social Media Platforms](https://mlsanity.com/paper/2211.08653/) (2022)
- [Dynamic Named Entity Recognition](https://mlsanity.com/paper/2302.10314/) (2023)
- [CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systems](https://mlsanity.com/paper/2405.15110/) (2024)
- [Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks](https://mlsanity.com/paper/2602.12759/) (2026)

## Abstract (arXiv)

In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state-of-the-art models we tested show such a bias; BERT fine-tuned models significantly outperforming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learnable adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and parameter freezing, leads to further gains.