# Adversarial Self-Supervised Data-Free Distillation for Text Classification

arXiv: https://arxiv.org/abs/2010.04883 · HTML: https://mlsanity.com/paper/2010.04883/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2020.

## Built on by (most cited first, 13 of 13)

- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt](https://mlsanity.com/paper/2205.07523/) (2022)
- [When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario](https://mlsanity.com/paper/2305.10013/) (2023)
- [Encapsulating Knowledge in One Prompt](https://mlsanity.com/paper/2407.11902/) (2024)
- [Up to 100× Faster Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.06253/) (2021)
- [Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classification](https://mlsanity.com/paper/2303.07643/) (2023)
- [Data-Free Distillation of Language Model by Text-to-Text Transfer](https://mlsanity.com/paper/2311.01689/) (2023)
- [Self-Regulated Data-Free Knowledge Amalgamation for Text Classification](https://mlsanity.com/paper/2406.15476/) (2024)
- [Can depth-adaptive BERT perform better on binary classification tasks](https://mlsanity.com/paper/2111.10951/) (2021)
- [Prompt Tuning for Few-Shot Continual Learning Named Entity Recognition](https://mlsanity.com/paper/2508.07248/) (2025)

## Abstract (arXiv)

Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computationally expensive model to a resource-efficient lightweight model. However, most KD algorithms, especially in NLP, rely on the accessibility of the original training dataset, which may be unavailable due to privacy issues. To tackle this problem, we propose a novel two-stage data-free distillation method, named Adversarial self-Supervised Data-Free Distillation (AS-DFD), which is designed for compressing large-scale transformer-based models (e.g., BERT). To avoid text generation in discrete space, we introduce a Plug & Play Embedding Guessing method to craft pseudo embeddings from the teacher's hidden knowledge. Meanwhile, with a self-supervised module to quantify the student's ability, we adapt the difficulty of pseudo embeddings in an adversarial training manner. To the best of our knowledge, our framework is the first data-free distillation framework designed for NLP tasks. We verify the effectiveness of our method on several text classification datasets.