# What's in a Name? Are BERT Named Entity Representations just as Good for any other Name?

arXiv: https://arxiv.org/abs/2007.06897 · HTML: https://mlsanity.com/paper/2007.06897/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 17 of 17)

- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lens](https://mlsanity.com/paper/2210.14011/) (2022)
- [Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition](https://mlsanity.com/paper/2107.11610/) (2021)
- [On the Robustness of Reading Comprehension Models to Entity Renaming](https://mlsanity.com/paper/2110.08555/) (2021)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering](https://mlsanity.com/paper/2212.10696/) (2022)
- [Breaking BERT: Understanding its Vulnerabilities for Named Entity Recognition through Adversarial Attack](https://mlsanity.com/paper/2109.11308/) (2021)
- [Can Language Representation Models Think in Bets?](https://mlsanity.com/paper/2210.07519/) (2022)
- [Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?](https://mlsanity.com/paper/2403.02009/) (2024)
- [NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation](https://mlsanity.com/paper/2111.05196/) (2021)
- [Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probes](https://mlsanity.com/paper/2210.01258/) (2022)
- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)
- [EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries](https://mlsanity.com/paper/2402.13372/) (2024)
- [Novel Aficionados and Doppelg"angers: a referential task for semantic representations of individual entities](https://mlsanity.com/paper/2104.10270/) (2021)
- [Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations](https://mlsanity.com/paper/2506.21682/) (2025)
- [From Insight to Exploit: Leveraging LLM Collaboration for Adaptive Adversarial Text Generation](https://mlsanity.com/paper/2511.03128/) (2025)

## Abstract (arXiv)

We evaluate named entity representations of BERT-based NLP models by investigating their robustness to replacements from the same typed class in the input. We highlight that on several tasks while such perturbations are natural, state of the art trained models are surprisingly brittle. The brittleness continues even with the recent entity-aware BERT models. We also try to discern the cause of this non-robustness, considering factors such as tokenization and frequency of occurrence. Then we provide a simple method that ensembles predictions from multiple replacements while jointly modeling the uncertainty of type annotations and label predictions. Experiments on three NLP tasks show that our method enhances robustness and increases accuracy on both natural and adversarial datasets.