# Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble

arXiv: https://arxiv.org/abs/2212.06522 · HTML: https://mlsanity.com/paper/2212.06522/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 13 Dec 2022.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Unified Named Entity Recognition as Word-Word Relation Classification](https://mlsanity.com/paper/2112.10070/) (2021)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [CompRess: Self-Supervised Learning by Compressing Representations](https://mlsanity.com/paper/2010.14713/) (2020)
- [Improving Coreference Resolution by Learning Entity-Level Distributed Representations](https://mlsanity.com/paper/1606.01323/) (2016)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition](https://mlsanity.com/paper/2012.05426/) (2020)
- [Low-Resource Name Tagging Learned with Weakly Labeled Data](https://mlsanity.com/paper/1908.09659/) (2019)
- [Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning](https://mlsanity.com/paper/2204.09589/) (2022)
- [Read, Retrospect, Select: An MRC Framework to Short Text Entity Linking](https://mlsanity.com/paper/2101.02394/) (2021)
- [Efficient Document-level Event Extraction via Pseudo-Trigger-aware Pruned Complete Graph](https://mlsanity.com/paper/2112.06013/) (2021)
- [Improving Distantly-Supervised Named Entity Recognition with Self-Collaborative Denoising Learning](https://mlsanity.com/paper/2110.04429/) (2021)
- [Delving Deep into Regularity: A Simple but Effective Method for Chinese Named Entity Recognition](https://mlsanity.com/paper/2204.05544/) (2022)

## Built on by (most cited first, 14 of 14)

- [TREA: Tree-Structure Reasoning Schema for Conversational Recommendation](https://mlsanity.com/paper/2307.10543/) (2023)
- [Mirror: A Universal Framework for Various Information Extraction Tasks](https://mlsanity.com/paper/2311.05419/) (2023)
- [MIRACLE: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control](https://mlsanity.com/paper/2310.18342/) (2023)
- [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023)
- [Joint Multi-Facts Reasoning Network For Complex Temporal Question Answering Over Knowledge Graph](https://mlsanity.com/paper/2401.02212/) (2024)
- [Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach](https://mlsanity.com/paper/2402.14948/) (2024)
- [Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language](https://mlsanity.com/paper/2605.29793/) (2026)
- [Enhancing Low-Resource Relation Representations through Multi-View Decoupling](https://mlsanity.com/paper/2312.17267/) (2023)
- [Deep Learning Based Named Entity Recognition Models for Recipes](https://mlsanity.com/paper/2402.17447/) (2024)
- [CogMG: Collaborative Augmentation Between Large Language Model and Knowledge Graph](https://mlsanity.com/paper/2406.17231/) (2024)
- [Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning](https://mlsanity.com/paper/2311.08010/) (2023)
- [Improving Pseudo Labels with Global-Local Denoising Framework for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2406.01213/) (2024)
- [Enhancing AI Research Paper Analysis: Methodology Component Extraction using Factored Transformer-based Sequence Modeling Approach](https://mlsanity.com/paper/2311.03401/) (2023)
- [Unsupervised Multidomain Approaches to Named Entity Recognition with Small Datasets](https://mlsanity.com/paper/2608.00984/) (2026)

## Abstract (arXiv)

Distantly-Supervised Named Entity Recognition (DS-NER) effectively alleviates the data scarcity problem in NER by automatically generating training samples. Unfortunately, the distant supervision may induce noisy labels, thus undermining the robustness of the learned models and restricting the practical application. To relieve this problem, recent works adopt self-training teacher-student frameworks to gradually refine the training labels and improve the generalization ability of NER models. However, we argue that the performance of the current self-training frameworks for DS-NER is severely underestimated by their plain designs, including both inadequate student learning and coarse-grained teacher updating. Therefore, in this paper, we make the first attempt to alleviate these issues by proposing: (1) adaptive teacher learning comprised of joint training of two teacher-student networks and considering both consistent and inconsistent predictions between two teachers, thus promoting comprehensive student learning. (2) fine-grained student ensemble that updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise. To verify the effectiveness of our proposed method, we conduct experiments on four DS-NER datasets. The experimental results demonstrate that our method significantly surpasses previous SOTA methods.