# WIDER & CLOSER: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition

arXiv: https://arxiv.org/abs/2212.03506 · HTML: https://mlsanity.com/paper/2212.03506/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [Adversarial Training for Large Neural Language Models](https://mlsanity.com/paper/2004.08994/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [Neural Cross-Lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1808.09861/) (2018)
- [Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language](https://mlsanity.com/paper/2004.12440/) (2020)
- [Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection](https://mlsanity.com/paper/1707.02483/) (2017)
- [Entity Projection via Machine Translation for Cross-Lingual NER](https://mlsanity.com/paper/1909.05356/) (2019)
- [Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER](https://mlsanity.com/paper/1909.00153/) (2019)
- [UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data](https://mlsanity.com/paper/2007.07683/) (2020)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [MultiCoNER: A Large-scale Multilingual dataset for Complex Named Entity Recognition](https://mlsanity.com/paper/2208.14536/) (2022)
- [Zero-Resource Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2002.05923/) (2020)
- [AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER](https://mlsanity.com/paper/2106.02300/) (2021)
- [Neural Adaptation Layers for Cross-domain Named Entity Recognition](https://mlsanity.com/paper/1810.06368/) (2018)
- [Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2106.00241/) (2021)
- [Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation](https://mlsanity.com/paper/2005.07839/) (2020)
- [Hybrid semi-Markov CRF for Neural Sequence Labeling](https://mlsanity.com/paper/1805.03838/) (2018)
- [USTC-NELSLIP at SemEval-2022 Task 11: Gazetteer-Adapted Integration Network for Multilingual Complex Named Entity Recognition](https://mlsanity.com/paper/2203.03216/) (2022)
- [Domain-Invariant Feature Distillation for Cross-Domain Sentiment Classification](https://mlsanity.com/paper/1908.09122/) (2019)
- [Target-Oriented Fine-tuning for Zero-Resource Named Entity Recognition](https://mlsanity.com/paper/2107.10523/) (2021)

## Built on by (most cited first, 7 of 7)

- [Neighboring Perturbations of Knowledge Editing on Large Language Models](https://mlsanity.com/paper/2401.17623/) (2024)
- [On the use of Silver Standard Data for Zero-shot Classification Tasks in Information Extraction](https://mlsanity.com/paper/2402.18061/) (2024)
- [USTC-NELSLIP at SemEval-2023 Task 2: Statistical Construction and Dual Adaptation of Gazetteer for Multilingual Complex NER](https://mlsanity.com/paper/2305.02517/) (2023)
- [Improving Pseudo Labels with Global-Local Denoising Framework for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2406.01213/) (2024)
- [SHINE: Syntax-augmented Hierarchical Interactive Encoder for Zero-shot Cross-lingual Information Extraction](https://mlsanity.com/paper/2305.12389/) (2023)
- [Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages](https://mlsanity.com/paper/2307.08714/) (2023)
- [Zero-shot Cross-lingual NER via Mitigating Language Difference: An Entity-aligned Translation Perspective](https://mlsanity.com/paper/2509.01147/) (2025)

## Abstract (arXiv)

Zero-shot cross-lingual named entity recognition (NER) aims at transferring knowledge from annotated and rich-resource data in source languages to unlabeled and lean-resource data in target languages. Existing mainstream methods based on the teacher-student distillation framework ignore the rich and complementary information lying in the intermediate layers of pre-trained language models, and domain-invariant information is easily lost during transfer. In this study, a mixture of short-channel distillers (MSD) method is proposed to fully interact the rich hierarchical information in the teacher model and to transfer knowledge to the student model sufficiently and efficiently. Concretely, a multi-channel distillation framework is designed for sufficient information transfer by aggregating multiple distillers as a mixture. Besides, an unsupervised method adopting parallel domain adaptation is proposed to shorten the channels between the teacher and student models to preserve domain-invariant features. Experiments on four datasets across nine languages demonstrate that the proposed method achieves new state-of-the-art performance on zero-shot cross-lingual NER and shows great generalization and compatibility across languages and fields.