# XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

arXiv: https://arxiv.org/abs/2004.05686 · HTML: https://mlsanity.com/paper/2004.05686/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2020.

## Built on by (most cited first, 30 of 38)

- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [AgentInstruct: Toward Generative Teaching with Agentic Flows](https://mlsanity.com/paper/2407.03502/) (2024)
- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models](https://mlsanity.com/paper/2212.10670/) (2022)
- [Joint Prompt Optimization of Stacked LLMs using Variational Inference](https://mlsanity.com/paper/2306.12509/) (2023)
- [Efficient Knowledge Injection in LLMs via Self-Distillation](https://mlsanity.com/paper/2412.14964/) (2024)
- [Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Rates](https://mlsanity.com/paper/2506.12459/) (2025)
- [XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation](https://mlsanity.com/paper/2106.04563/) (2021)
- [AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER](https://mlsanity.com/paper/2106.02300/) (2021)
- [No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval](https://mlsanity.com/paper/2206.02873/) (2022)
- [Learning Cross-Lingual IR from an English Retriever](https://mlsanity.com/paper/2112.08185/) (2021)
- [A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training](https://mlsanity.com/paper/2305.02031/) (2023)
- [MetaQA: Combining Expert Agents for Multi-Skill Question Answering](https://mlsanity.com/paper/2112.01922/) (2021)
- [Intriguing Properties of Compression on Multilingual Models](https://mlsanity.com/paper/2211.02738/) (2022)
- [Efficient Image Captioning for Edge Devices](https://mlsanity.com/paper/2212.08985/) (2022)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crisis Response](https://mlsanity.com/paper/2210.04573/) (2022)
- [From Big to Small Without Losing It All: Text Augmentation with ChatGPT for Efficient Sentiment Analysis](https://mlsanity.com/paper/2312.04720/) (2023)
- [LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation](https://mlsanity.com/paper/2103.06418/) (2021)
- [FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks](https://mlsanity.com/paper/2110.11869/) (2021)
- [Efficiently Distilling LLMs for Edge Applications](https://mlsanity.com/paper/2404.01353/) (2024)
- [CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation](https://mlsanity.com/paper/2204.07674/) (2022)
- [EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics](https://mlsanity.com/paper/2209.09593/) (2022)
- [Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems](https://mlsanity.com/paper/2201.05767/) (2022)
- [Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?](https://mlsanity.com/paper/2301.12609/) (2023)
- [Distillation of encoder-decoder transformers for sequence labelling](https://mlsanity.com/paper/2302.05454/) (2023)
- [Semantic Retrieval at Walmart](https://mlsanity.com/paper/2412.04637/) (2024)
- [AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer](https://mlsanity.com/paper/2505.16463/) (2025)
- [Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)](https://mlsanity.com/paper/2501.05465/) (2025)

## Abstract (arXiv)

Deep and large pre-trained language models are the state-of-the-art for various natural language processing tasks. However, the huge size of these models could be a deterrent to use them in practice. Some recent and concurrent works use knowledge distillation to compress these huge models into shallow ones. In this work we study knowledge distillation with a focus on multi-lingual Named Entity Recognition (NER). In particular, we study several distillation strategies and propose a stage-wise optimization scheme leveraging teacher internal representations that is agnostic of teacher architecture and show that it outperforms strategies employed in prior works. Additionally, we investigate the role of several factors like the amount of unlabeled data, annotation resources, model architecture and inference latency to name a few. We show that our approach leads to massive compression of MBERT-like teacher models by upto 35x in terms of parameters and 51x in terms of latency for batch inference while retaining 95% of its F1-score for NER over 41 languages.