# Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition

arXiv: https://arxiv.org/abs/2106.00241 · HTML: https://mlsanity.com/paper/2106.00241/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [Translation Artifacts in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2004.04721/) (2020)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [A Study of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1808.08866/) (2018)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [Neural Cross-Lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1808.09861/) (2018)
- [Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System](https://mlsanity.com/paper/1910.08381/) (2019)
- [Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language](https://mlsanity.com/paper/2004.12440/) (2020)
- [Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection](https://mlsanity.com/paper/1707.02483/) (2017)
- [Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks](https://mlsanity.com/paper/1703.06345/) (2017)
- [Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1911.06161/) (2019)
- [Zero-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/1911.09812/) (2019)
- [A User-Centered Concept Mining System for Query and Document Understanding at Tencent](https://mlsanity.com/paper/1905.08487/) (2019)
- [Towards Lingua Franca Named Entity Recognition with BERT](https://mlsanity.com/paper/1912.01389/) (2019)
- [Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension](https://mlsanity.com/paper/2004.14069/) (2020)
- [Cross-lingual Distillation for Text Classification](https://mlsanity.com/paper/1705.02073/) (2017)
- [Do Explicit Alignments Robustly Improve Multilingual Encoders?](https://mlsanity.com/paper/2010.02537/) (2020)
- [Mining Implicit Relevance Feedback from User Behavior for Web Question Answering](https://mlsanity.com/paper/2006.07581/) (2020)
- [On the Importance of Word Order Information in Cross-lingual Sequence Labeling](https://mlsanity.com/paper/2001.11164/) (2020)

## Built on by (most cited first, 15 of 15)

- [FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2206.06561/) (2022)
- [StructVPR: Distill Structural Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2212.00937/) (2022)
- [ConNER: Consistency Training for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2211.09394/) (2022)
- [Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning](https://mlsanity.com/paper/2305.13628/) (2023)
- [Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling](https://mlsanity.com/paper/2204.05210/) (2022)
- [WIDER & CLOSER: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2212.03506/) (2022)
- [StructVPR++: Distill Structural and Semantic Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2503.06601/) (2025)
- [A Dual-Contrastive Framework for Low-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2204.00796/) (2022)
- [ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2301.08855/) (2023)
- [Improving Pseudo Labels with Global-Local Denoising Framework for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2406.01213/) (2024)
- [SHINE: Syntax-augmented Hierarchical Interactive Encoder for Zero-shot Cross-lingual Information Extraction](https://mlsanity.com/paper/2305.12389/) (2023)
- [CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2305.14913/) (2023)
- [Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation](https://mlsanity.com/paper/2307.00534/) (2023)
- [DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2403.04158/) (2024)
- [Zero-shot Cross-lingual NER via Mitigating Language Difference: An Entity-aligned Translation Perspective](https://mlsanity.com/paper/2509.01147/) (2025)

## Abstract (arXiv)

Named entity recognition (NER) is a fundamental component in many applications, such as Web Search and Voice Assistants. Although deep neural networks greatly improve the performance of NER, due to the requirement of large amounts of training data, deep neural networks can hardly scale out to many languages in an industry setting. To tackle this challenge, cross-lingual NER transfers knowledge from a rich-resource language to languages with low resources through pre-trained multilingual language models. Instead of using training data in target languages, cross-lingual NER has to rely on only training data in source languages, and optionally adds the translated training data derived from source languages. However, the existing cross-lingual NER methods do not make good use of rich unlabeled data in target languages, which is relatively easy to collect in industry applications. To address the opportunities and challenges, in this paper we describe our novel practice in Microsoft to leverage such large amounts of unlabeled data in target languages in real production settings. To effectively extract weak supervision signals from the unlabeled data, we develop a novel approach based on the ideas of semi-supervised learning and reinforcement learning. The empirical study on three benchmark data sets verifies that our approach establishes the new state-of-the-art performance with clear edges. Now, the NER techniques reported in this paper are on their way to become a fundamental component for Web ranking, Entity Pane, Answers Triggering, and Question Answering in the Microsoft Bing search engine. Moreover, our techniques will also serve as part of the Spoken Language Understanding module for a commercial voice assistant. We plan to open source the code of the prototype framework after deployment.