# Data Distillation for Text Classification

arXiv: https://arxiv.org/abs/2104.08448 · HTML: https://mlsanity.com/paper/2104.08448/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Flexible Dataset Distillation: Learn Labels Instead of Images](https://mlsanity.com/paper/2006.08572/) (2020)
- [Neural Compatibility Modeling with Attentive Knowledge Distillation](https://mlsanity.com/paper/1805.00313/) (2018)
- [Explicit Interaction Model towards Text Classification](https://mlsanity.com/paper/1811.09386/) (2018)

## Built on by (most cited first, 25 of 25)

- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Accelerating Dataset Distillation via Model Augmentation](https://mlsanity.com/paper/2212.06152/) (2022)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement](https://mlsanity.com/paper/2312.00362/) (2023)
- [DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation](https://mlsanity.com/paper/2404.00264/) (2024)
- [Low-Rank Similarity Mining for Multimodal Dataset Distillation](https://mlsanity.com/paper/2406.03793/) (2024)
- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation](https://mlsanity.com/paper/2502.02854/) (2025)
- [Synthetic Text Generation for Training Large Language Models via Gradient Matching](https://mlsanity.com/paper/2502.17607/) (2025)
- [Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matching](https://mlsanity.com/paper/2402.04924/) (2024)
- [UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation](https://mlsanity.com/paper/2504.20500/) (2025)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments](https://mlsanity.com/paper/2312.15842/) (2023)
- [On Learning Representations for Tabular Data Distillation](https://mlsanity.com/paper/2501.13905/) (2025)
- [Simple yet Effective Graph Distillation via Clustering](https://mlsanity.com/paper/2505.20807/) (2025)
- [Video Set Distillation: Information Diversification and Temporal Densification](https://mlsanity.com/paper/2412.00111/) (2024)
- [CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder](https://mlsanity.com/paper/2510.18583/) (2025)
- [What's New in My Data? Novelty Exploration via Contrastive Generation](https://mlsanity.com/paper/2410.14765/) (2024)
- [Transferable text data distillation by trajectory matching](https://mlsanity.com/paper/2504.09818/) (2025)
- [Multimodal Dataset Distillation via Phased Teacher Models](https://mlsanity.com/paper/2603.25388/) (2026)
- [TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation](https://mlsanity.com/paper/2607.11898/) (2026)
- [Dataset Distillation of 3D Point Clouds via Distribution Matching](https://mlsanity.com/paper/2503.22154/) (2025)
- [TaLK: Text-attributed Graph Dataset Distillation via Coupling Language Model with Graph-Aware Kernel](https://mlsanity.com/paper/2606.22975/) (2026)
- [Semi-Supervised Text-Attributed Graph Distillation](https://mlsanity.com/paper/2607.20477/) (2026)

## Abstract (arXiv)

Deep learning techniques have achieved great success in many fields, while at the same time deep learning models are getting more complex and expensive to compute. It severely hinders the wide applications of these models. In order to alleviate this problem, model distillation emerges as an effective means to compress a large model into a smaller one without a significant drop in accuracy. In this paper, we study a related but orthogonal issue, data distillation, which aims to distill the knowledge from a large training dataset down to a smaller and synthetic one. It has the potential to address the large and growing neural network training problem based on the small dataset. We develop a novel data distillation method for text classification. We evaluate our method on eight benchmark datasets. The results that the distilled data with the size of 0.1% of the original text data achieves approximately 90% performance of the original is rather impressive.