# Towards Zero-Shot Knowledge Distillation for Natural Language Processing

arXiv: https://arxiv.org/abs/2012.15495 · HTML: https://mlsanity.com/paper/2012.15495/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 20 of 20)

- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models](https://mlsanity.com/paper/2212.10670/) (2022)
- [The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey](https://mlsanity.com/paper/2401.07872/) (2024)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation](https://mlsanity.com/paper/2105.05912/) (2021)
- [RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation](https://mlsanity.com/paper/2109.10164/) (2021)
- [Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher](https://mlsanity.com/paper/2110.08532/) (2021)
- [A Short Study on Compressing Decoder-Based Language Models](https://mlsanity.com/paper/2110.08460/) (2021)
- [Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax](https://mlsanity.com/paper/2105.13608/) (2021)
- [Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt](https://mlsanity.com/paper/2205.07523/) (2022)
- [Improving Question Answering Performance Using Knowledge Distillation and Active Learning](https://mlsanity.com/paper/2109.12662/) (2021)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding](https://mlsanity.com/paper/2109.05696/) (2021)
- [CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation](https://mlsanity.com/paper/2204.07674/) (2022)
- [Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization](https://mlsanity.com/paper/2212.05998/) (2022)
- [Knowledge Distillation with Noisy Labels for Natural Language Understanding](https://mlsanity.com/paper/2109.10147/) (2021)
- [Data-Free Distillation of Language Model by Text-to-Text Transfer](https://mlsanity.com/paper/2311.01689/) (2023)
- [A Novel Compression Framework for YOLOv8: Achieving Real-Time Aerial Object Detection on Edge Devices via Structured Pruning and Channel-Wise Distillation](https://mlsanity.com/paper/2509.12918/) (2025)
- [Enriching Knowledge Distillation with Intra-Class Contrastive Learning](https://mlsanity.com/paper/2509.22053/) (2025)

## Abstract (arXiv)

Knowledge Distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. In its regular manifestations, KD requires access to the teacher's training data for knowledge transfer to the student network. However, privacy concerns, data regulations and proprietary reasons may prevent access to such data. We present, to the best of our knowledge, the first work on Zero-Shot Knowledge Distillation for NLP, where the student learns from the much larger teacher without any task specific data. Our solution combines out of domain data and adversarial training to learn the teacher's output distribution. We investigate six tasks from the GLUE benchmark and demonstrate that we can achieve between 75% and 92% of the teacher's classification score (accuracy or F1) while compressing the model 30 times.