# Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

arXiv: https://arxiv.org/abs/2109.05003 · HTML: https://mlsanity.com/paper/2109.05003/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification](https://mlsanity.com/paper/2004.12239/) (2020)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [An Analysis of Simple Data Augmentation for Named Entity Recognition](https://mlsanity.com/paper/2010.11683/) (2020)
- [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [Neural Cross-Lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1808.09861/) (2018)
- [Weakly-Supervised Hierarchical Text Classification](https://mlsanity.com/paper/1812.11270/) (2018)
- [Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection](https://mlsanity.com/paper/1707.02483/) (2017)
- [Learning to Denoise Distantly-Labeled Data for Entity Typing](https://mlsanity.com/paper/1905.01566/) (2019)

## Built on by (most cited first, 30 of 41)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER](https://mlsanity.com/paper/2110.08454/) (2021)
- [Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning](https://mlsanity.com/paper/2211.03044/) (2022)
- [Language Models as Inductive Reasoners](https://mlsanity.com/paper/2212.10923/) (2022)
- [Prompt Perturbation Consistency Learning for Robust Language Models](https://mlsanity.com/paper/2402.15833/) (2024)
- [Domain-Specific NER via Retrieving Correlated Samples](https://mlsanity.com/paper/2208.12995/) (2022)
- [Traffic-Domain Video Question Answering with Automatic Captioning](https://mlsanity.com/paper/2307.09636/) (2023)
- [Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents](https://mlsanity.com/paper/2203.02898/) (2022)
- [Few-Shot Fine-Grained Entity Typing with Automatic Label Interpretation and Instance Generation](https://mlsanity.com/paper/2206.13746/) (2022)
- [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training](https://mlsanity.com/paper/2305.13723/) (2023)
- [CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning](https://mlsanity.com/paper/2411.00873/) (2024)
- [Simple Questions Generate Named Entity Recognition Datasets](https://mlsanity.com/paper/2112.08808/) (2021)
- [General-to-Specific Transfer Labeling for Domain Adaptable Keyphrase Generation](https://mlsanity.com/paper/2208.09606/) (2022)
- [Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach](https://mlsanity.com/paper/2402.14948/) (2024)
- [PInKS: Preconditioned Commonsense Inference with Minimal Supervision](https://mlsanity.com/paper/2206.07920/) (2022)
- [SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.04076/) (2023)
- [MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2310.08298/) (2023)
- [Improving Distantly Supervised Relation Extraction by Natural Language Inference](https://mlsanity.com/paper/2208.00346/) (2022)
- [Biomedical Named Entity Recognition via Dictionary-based Synonym Generalization](https://mlsanity.com/paper/2305.13066/) (2023)
- [Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training](https://mlsanity.com/paper/2305.12634/) (2023)
- [Better Sampling of Negatives for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.13142/) (2023)
- [Distantly-Supervised Joint Extraction with Noise-Robust Learning](https://mlsanity.com/paper/2310.04994/) (2023)
- [Granular-ball Representation Learning for Deep CNN on Learning with Label Noise](https://mlsanity.com/paper/2409.03254/) (2024)
- [DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2504.04616/) (2025)
- [Automatic Creation of Named Entity Recognition Datasets by Querying Phrase Representations](https://mlsanity.com/paper/2210.07586/) (2022)
- [Demonstration-based learning for few-shot biomedical named entity recognition under machine reading comprehension](https://mlsanity.com/paper/2308.06454/) (2023)
- [ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models](https://mlsanity.com/paper/2403.11103/) (2024)
- [How to tackle an emerging topic? Combining strong and weak labels for Covid news NER](https://mlsanity.com/paper/2209.15108/) (2022)

## Abstract (arXiv)

We study the problem of training named entity recognition (NER) models using only distantly-labeled data, which can be automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base. The biggest challenge of distantly-supervised NER is that the distant supervision may induce incomplete and noisy labels, rendering the straightforward application of supervised learning ineffective. In this paper, we propose (1) a noise-robust learning scheme comprised of a new loss function and a noisy label removal step, for training NER models on distantly-labeled data, and (2) a self-training method that uses contextualized augmentations created by pre-trained language models to improve the generalization ability of the NER model. On three benchmark datasets, our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.