# NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging

arXiv: https://arxiv.org/abs/2112.00405 · HTML: https://mlsanity.com/paper/2112.00405/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling](https://mlsanity.com/paper/2004.11727/) (2020)
- [Low-Resource Name Tagging Learned with Weakly Labeled Data](https://mlsanity.com/paper/1908.09659/) (2019)
- [Zero-Resource Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2002.05923/) (2020)

## Built on by (most cited first, 21 of 21)

- [NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data](https://mlsanity.com/paper/2402.15343/) (2024)
- [One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER](https://mlsanity.com/paper/2301.10410/) (2023)
- [A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER](https://mlsanity.com/paper/2308.14533/) (2023)
- [General-to-Specific Transfer Labeling for Domain Adaptable Keyphrase Generation](https://mlsanity.com/paper/2208.09606/) (2022)
- [Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Model](https://mlsanity.com/paper/2411.00451/) (2024)
- [PerPLM: Personalized Fine-tuning of Pretrained Language Models via Writer-specific Intermediate Learning and Prompts](https://mlsanity.com/paper/2309.07727/) (2023)
- [Dealing with Annotator Disagreement in Hate Speech Classification](https://mlsanity.com/paper/2502.08266/) (2025)
- [From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper](https://mlsanity.com/paper/2503.07450/) (2025)
- [Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data](https://mlsanity.com/paper/2404.05632/) (2024)
- [Small Language Models in the Real World: Insights from Industrial Text Classification](https://mlsanity.com/paper/2505.16078/) (2025)
- [Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection](https://mlsanity.com/paper/2511.21066/) (2025)
- [Effective Transfer Learning for Low-Resource Natural Language Understanding](https://mlsanity.com/paper/2208.09180/) (2022)
- [Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledge](https://mlsanity.com/paper/2210.07523/) (2022)
- [Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows](https://mlsanity.com/paper/2411.19295/) (2024)
- [MQDD: Pre-training of Multimodal Question Duplicity Detection for Software Engineering Domain](https://mlsanity.com/paper/2203.14093/) (2022)
- [Adversarial Adaptation for French Named Entity Recognition](https://mlsanity.com/paper/2301.05220/) (2023)
- [NER-to-MRC: Named-Entity Recognition Completely Solving as Machine Reading Comprehension](https://mlsanity.com/paper/2305.03970/) (2023)
- [DistALANER: Distantly Supervised Active Learning Augmented Named Entity Recognition in the Open Source Software Ecosystem](https://mlsanity.com/paper/2402.16159/) (2024)
- [Improve Academic Query Resolution through BERT-based Question Extraction from Images](https://mlsanity.com/paper/2405.01587/) (2024)
- [RATE: An LLM-Powered Retrieval Augmented Generation Technology-Extraction Pipeline](https://mlsanity.com/paper/2507.21125/) (2025)
- [Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER](https://mlsanity.com/paper/2510.07566/) (2025)

## Abstract (arXiv)

Named entity recognition (NER) models generally perform poorly when large training datasets are unavailable for low-resource domains. Recently, pre-training a large-scale language model has become a promising direction for coping with the data scarcity issue. However, the underlying discrepancies between the language modeling and NER task could limit the models' performance, and pre-training for the NER task has rarely been studied since the collected NER datasets are generally small or large but with low quality. In this paper, we construct a massive NER corpus with a relatively high quality, and we pre-train a NER-BERT model based on the created dataset. Experimental results show that our pre-trained model can significantly outperform BERT as well as other strong baselines in low-resource scenarios across nine diverse domains. Moreover, a visualization of entity representations further indicates the effectiveness of NER-BERT for categorizing a variety of entities.