# Structure-Level Knowledge Distillation For Multilingual Sequence Labeling

arXiv: https://arxiv.org/abs/2004.03846 · HTML: https://mlsanity.com/paper/2004.03846/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation](https://mlsanity.com/paper/2406.16524/) (2024)
- [Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model](https://mlsanity.com/paper/2211.01200/) (2022)
- [Robust Interaction-Based Relevance Modeling for Online e-Commerce Search](https://mlsanity.com/paper/2406.02135/) (2024)
- [Efficient Sub-structured Knowledge Distillation](https://mlsanity.com/paper/2203.04825/) (2022)
- [How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?](https://mlsanity.com/paper/2212.10767/) (2022)
- [Can depth-adaptive BERT perform better on binary classification tasks](https://mlsanity.com/paper/2111.10951/) (2021)
- [Unraveling Key Factors of Knowledge Distillation](https://mlsanity.com/paper/2312.08585/) (2023)
- [An Aspect Extraction Framework using Different Embedding Types, Learning Models, and Dependency Structure](https://mlsanity.com/paper/2503.03512/) (2025)
- [Evolving Knowledge Distillation for Lightweight Neural Machine Translation](https://mlsanity.com/paper/2605.09924/) (2026)

## Abstract (arXiv)

Multilingual sequence labeling is a task of predicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, using a multilingual model has the benefit of a smaller model size, easier in online serving, and generalizability to low-resource languages. However, current multilingual models still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance between the student's and the teachers' structure level probability distributions, (2) aggregates the structure-level knowledge to local distributions and minimizes the distance between two local probability distributions. Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.