# Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data

arXiv: https://arxiv.org/abs/2106.08977 · HTML: https://mlsanity.com/paper/2106.08977/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Self-training Improves Pre-training for Natural Language Understanding](https://mlsanity.com/paper/2010.02194/) (2020)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data](https://mlsanity.com/paper/1704.06360/) (2017)
- [Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling](https://mlsanity.com/paper/1910.04289/) (2019)
- [Adaptive Self-training for Few-shot Neural Sequence Labeling](https://mlsanity.com/paper/2010.03680/) (2020)
- [Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data](https://mlsanity.com/paper/2010.11506/) (2020)
- [Unsupervised Aspect Term Extraction with B-LSTM & CRF using Automatically Labelled Datasets](https://mlsanity.com/paper/1709.05094/) (2017)

## Built on by (most cited first, 18 of 18)

- [Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT](https://mlsanity.com/paper/2308.06294/) (2023)
- [Few-shot Named Entity Recognition with Self-describing Networks](https://mlsanity.com/paper/2203.12252/) (2022)
- [DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding](https://mlsanity.com/paper/2112.01047/) (2021)
- [QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction](https://mlsanity.com/paper/2108.08468/) (2021)
- [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023)
- [Weaker Than You Think: A Critical Look at Weakly Supervised Learning](https://mlsanity.com/paper/2305.17442/) (2023)
- [Self-Training with Differentiable Teacher](https://mlsanity.com/paper/2109.07049/) (2021)
- [Ontology-Driven and Weakly Supervised Rare Disease Identification from Clinical Notes](https://mlsanity.com/paper/2205.05656/) (2022)
- [Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding](https://mlsanity.com/paper/2210.03915/) (2022)
- [Biomedical NER for the Enterprise with Distillated BERN2 and the Kazu Framework](https://mlsanity.com/paper/2212.00223/) (2022)
- [Enhancing Low-resource Fine-grained Named Entity Recognition by Leveraging Coarse-grained Datasets](https://mlsanity.com/paper/2310.11715/) (2023)
- [ChatUIE: Exploring Chat-based Unified Information Extraction using Large Language Models](https://mlsanity.com/paper/2403.05132/) (2024)
- [Logits-Constrained Framework with RoBERTa for Ancient Chinese NER](https://mlsanity.com/paper/2505.02983/) (2025)
- [How to tackle an emerging topic? Combining strong and weak labels for Covid news NER](https://mlsanity.com/paper/2209.15108/) (2022)
- [Scalable and Weakly Supervised Bank Transaction Classification](https://mlsanity.com/paper/2305.18430/) (2023)
- [Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances](https://mlsanity.com/paper/2310.16790/) (2023)
- [Refining and Reusing Annotation Guidelines for LLM Annotation](https://mlsanity.com/paper/2605.20809/) (2026)
- [Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels](https://mlsanity.com/paper/2608.30827/) (2026)

## Abstract (arXiv)

Weak supervision has shown promising results in many natural language processing tasks, such as Named Entity Recognition (NER). Existing work mainly focuses on learning deep NER models only with weak supervision, i.e., without any human annotation, and shows that by merely using weakly labeled data, one can achieve good performance, though still underperforms fully supervised NER with manually/strongly labeled data. In this paper, we consider a more practical scenario, where we have both a small amount of strongly labeled data and a large amount of weakly labeled data. Unfortunately, we observe that weakly labeled data does not necessarily improve, or even deteriorate the model performance (due to the extensive noise in the weak labels) when we train deep NER models over a simple or weighted combination of the strongly labeled and weakly labeled data. To address this issue, we propose a new multi-stage computational framework -- NEEDLE with three essential ingredients: (1) weak label completion, (2) noise-aware loss function, and (3) final fine-tuning over the strongly labeled data. Through experiments on E-commerce query NER and Biomedical NER, we demonstrate that NEEDLE can effectively suppress the noise of the weak labels and outperforms existing methods. In particular, we achieve new SOTA F1-scores on 3 Biomedical NER datasets: BC5CDR-chem 93.74, BC5CDR-disease 90.69, NCBI-disease 92.28.