# Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

arXiv: https://arxiv.org/abs/2012.01266 · HTML: https://mlsanity.com/paper/2012.01266/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 30 of 33)

- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [CLLMs: Consistency Large Language Models](https://mlsanity.com/paper/2403.00835/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [BERT Learns to Teach: Knowledge Distillation with Meta Learning](https://mlsanity.com/paper/2106.04570/) (2021)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models](https://mlsanity.com/paper/2404.08763/) (2024)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Position: Key Claims in LLM Research Have a Long Tail of Footnotes](https://mlsanity.com/paper/2308.07120/) (2023)
- [One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models](https://mlsanity.com/paper/2310.09499/) (2023)
- [Search for Efficient Large Language Models](https://mlsanity.com/paper/2409.17372/) (2024)
- [DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models](https://mlsanity.com/paper/2504.15027/) (2025)
- [EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing](https://mlsanity.com/paper/2205.00258/) (2022)
- [Hierarchical Skip Decoding for Efficient Autoregressive Text Generation](https://mlsanity.com/paper/2403.14919/) (2024)
- [Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers](https://mlsanity.com/paper/2404.09326/) (2024)
- [Few-Shot Learning of Compact Models via Task-Specific Meta Distillation](https://mlsanity.com/paper/2210.09922/) (2022)
- [HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression](https://mlsanity.com/paper/2110.08551/) (2021)
- [A Hybrid Cross-Stage Coordination Pre-ranking Model for Online Recommendation Systems](https://mlsanity.com/paper/2502.10284/) (2025)
- [Learning to Teach with Student Feedback](https://mlsanity.com/paper/2109.04641/) (2021)
- [DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings](https://mlsanity.com/paper/2112.05638/) (2021)
- [MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU](https://mlsanity.com/paper/2408.08144/) (2024)
- [EvoP: Robust LLM Inference via Evolutionary Pruning](https://mlsanity.com/paper/2502.14910/) (2025)
- [PIP: Perturbation-based Iterative Pruning for Large Language Models](https://mlsanity.com/paper/2501.15278/) (2025)
- [Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning](https://mlsanity.com/paper/2510.13832/) (2025)
- [VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking](https://mlsanity.com/paper/2511.18692/) (2025)
- [Domain Private Transformers for Multi-Domain Dialog Systems](https://mlsanity.com/paper/2305.14208/) (2023)
- [Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method](https://mlsanity.com/paper/2306.06625/) (2023)
- [Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series](https://mlsanity.com/paper/2511.01354/) (2025)
- [Experts are all you need: A Composable Framework for Large Language Model Inference](https://mlsanity.com/paper/2511.22955/) (2025)
- [How Is Uncertainty Propagated in Knowledge Distillation?](https://mlsanity.com/paper/2601.18909/) (2026)

## Abstract (arXiv)

Pre-trained language models have been applied to various NLP tasks with considerable performance gains. However, the large model sizes, together with the long inference time, limit the deployment of such models in real-time applications. One line of model compression approaches considers knowledge distillation to distill large teacher models into small student models. Most of these studies focus on single-domain only, which ignores the transferable knowledge from other domains. We notice that training a teacher with transferable knowledge digested across domains can achieve better generalization capability to help knowledge distillation. Hence we propose a Meta-Knowledge Distillation (Meta-KD) framework to build a meta-teacher model that captures transferable knowledge across domains and passes such knowledge to students. Specifically, we explicitly force the meta-teacher to capture transferable knowledge at both instance-level and feature-level from multiple domains, and then propose a meta-distillation algorithm to learn single-domain student models with guidance from the meta-teacher. Experiments on public multi-domain NLP tasks show the effectiveness and superiority of the proposed Meta-KD framework. Further, we also demonstrate the capability of Meta-KD in the settings where the training data is scarce.