# Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study

arXiv: https://arxiv.org/abs/2001.03844 · HTML: https://mlsanity.com/paper/2001.03844/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2020.

## Built on by (most cited first, 30 of 35)

- [SpanNER: Named Entity Re-/Recognition as Span Prediction](https://mlsanity.com/paper/2106.00641/) (2021)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition](https://mlsanity.com/paper/2404.18848/) (2024)
- [Memorization and Generalization in Neural Code Intelligence Models](https://mlsanity.com/paper/2106.08704/) (2021)
- [MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective](https://mlsanity.com/paper/2204.04391/) (2022)
- [Searching for Optimal Subword Tokenization in Cross-domain NER](https://mlsanity.com/paper/2206.03352/) (2022)
- [Data Augmentation for Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2109.01758/) (2021)
- [A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning](https://mlsanity.com/paper/2105.09198/) (2021)
- [LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty](https://mlsanity.com/paper/2402.10573/) (2024)
- [CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset](https://mlsanity.com/paper/2310.16225/) (2023)
- [Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition](https://mlsanity.com/paper/2108.11857/) (2021)
- [NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval](https://mlsanity.com/paper/2310.14282/) (2023)
- [Disparity Between Batches as a Signal for Early Stopping](https://mlsanity.com/paper/2107.06665/) (2021)
- [PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling](https://mlsanity.com/paper/2208.11508/) (2022)
- [Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization](https://mlsanity.com/paper/2403.19317/) (2024)
- [How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?](https://mlsanity.com/paper/2101.00160/) (2021)
- [A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries](https://mlsanity.com/paper/2303.06944/) (2023)
- [Do LLMs Surpass Encoders for Biomedical NER?](https://mlsanity.com/paper/2504.00664/) (2025)
- [Regularization for Long Named Entity Recognition](https://mlsanity.com/paper/2104.07249/) (2021)
- [Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations](https://mlsanity.com/paper/2310.03518/) (2023)
- [TMR: Evaluating NER Recall on Tough Mentions](https://mlsanity.com/paper/2103.12312/) (2021)
- [Larger-Context Tagging: When and Why Does It Work?](https://mlsanity.com/paper/2104.04434/) (2021)
- [Leveraging Expert Guided Adversarial Augmentation For Improving Generalization in Named Entity Recognition](https://mlsanity.com/paper/2203.10693/) (2022)
- [Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification](https://mlsanity.com/paper/2205.02129/) (2022)
- [Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction](https://mlsanity.com/paper/2306.11386/) (2023)
- [Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction](https://mlsanity.com/paper/2109.12008/) (2021)
- [A Cross-institutional Evaluation on Breast Cancer Phenotyping NLP Algorithms on Electronic Health Records](https://mlsanity.com/paper/2303.08448/) (2023)
- [CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English](https://mlsanity.com/paper/2405.11865/) (2024)

## Abstract (arXiv)

While neural network-based models have achieved impressive performance on a large body of NLP tasks, the generalization behavior of different models remains poorly understood: Does this excellent performance imply a perfect generalization model, or are there still some limitations? In this paper, we take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives and characterize the differences of their generalization abilities through the lens of our proposed measures, which guides us to better design models and training methods. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models in terms of breakdown performance analysis, annotation errors, dataset bias, and category relationships, which suggest directions for improvement. We have released the datasets: (ReCoNLL, PLONER) for the future research at our project page: http://pfliu.com/InterpretNER/. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers and classifies them into different research topics: https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers.