mlsanityData to 7 Oct 2026

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

arXiv 2004.05686 ↗First posted 12 Apr 2020cs.CL cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Deep and large pre-trained language models are the state-of-the-art for various natural language processing tasks. However, the huge size of these models could be a deterrent to use them in practice. Some recent and concurrent works use knowledge distillation to compress these huge models into shallow ones. In this work we study knowledge distillation with a focus on multi-lingual Named Entity Recognition (NER). In particular, we study several distillation strategies and propose a stage-wise optimization scheme leveraging teacher internal representations that is agnostic of teacher architecture and show that it outperforms strategies employed in prior works. Additionally, we investigate the role of several factors like the amount of unlabeled data, annotation resources, model architecture and inference latency to name a few. We show that our approach leads to massive compression of MBERT-like teacher models by upto 35x in terms of parameters and 51x in terms of latency for batch inference while retaining 95% of its F1-score for NER over 41 languages.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 10; 2023: 7; 2024: 8; 2025: 5.

38 papers.

Sort:

By area:

  1. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  2. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  3. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  4. 2024A Survey on Transformer Compressioncited by 44
  5. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  6. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  7. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 22
  8. 2025Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Ratescited by 19
  9. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  10. 2021AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NERcited by 17
  11. 2022No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrievalcited by 17
  12. 2021Learning Cross-Lingual IR from an English Retrievercited by 16
26 more
  1. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  2. 2021MetaQA: Combining Expert Agents for Multi-Skill Question Answeringcited by 15
  3. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  4. 2022Efficient Image Captioning for Edge Devicescited by 14
  5. 2022Differentially Private Model Compressioncited by 13
  6. 2022HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crisis Responsecited by 7
  7. 2023From Big to Small Without Losing It All: Text Augmentation with ChatGPT for Efficient Sentiment Analysiscited by 6
  8. 2021LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillationcited by 5
  9. 2021FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networkscited by 5
  10. 2024Efficiently Distilling LLMs for Edge Applicationscited by 5
  11. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4
  12. 2022EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metricscited by 4
  13. 2022Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systemscited by 3
  14. 2023Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?cited by 3
  15. 2023Distillation of encoder-decoder transformers for sequence labellingcited by 3
  16. 2024Semantic Retrieval at Walmartcited by 3
  17. 2025AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformercited by 3
  18. 2025Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)cited by 2
  19. 2025Explaining Matters: Leveraging Definitions and Semantic Expansion for Sexism Detectioncited by 2
  20. 2021RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Modelscited by 1
  21. 2022Compressing Cross-Lingual Multi-Task Models at Qualtricscited by 1
  22. 2024Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increasecited by 1
  23. 2021Classification-based Quality Estimation: Small and Efficient Models for Real-world Applicationscited by 0
  24. 2023Friend-training: Learning from Models of Different but Related Taskscited by 0
  25. 2024An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillationcited by 0
  26. 2025Knowledge distillation through geometry-aware representational alignmentcited by 0