# Technical Report on Text Dataset Distillation

arXiv: https://arxiv.org/abs/2512.03967 · HTML: https://mlsanity.com/paper/2512.03967/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)
- [A Survey on Dataset Distillation: Approaches, Applications and Future Directions](https://mlsanity.com/paper/2305.01975/) (2023)
- [A Label is Worth a Thousand Images in Dataset Distillation](https://mlsanity.com/paper/2406.10485/) (2024)
- [DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation](https://mlsanity.com/paper/2404.00264/) (2024)
- [Multisize Dataset Condensation](https://mlsanity.com/paper/2403.06075/) (2024)
- [Low-Rank Similarity Mining for Multimodal Dataset Distillation](https://mlsanity.com/paper/2406.03793/) (2024)
- [Synthetic Text Generation for Training Large Language Models via Gradient Matching](https://mlsanity.com/paper/2502.17607/) (2025)
- [UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation](https://mlsanity.com/paper/2504.20500/) (2025)

## Abstract (arXiv)

In the vision domain, dataset distillation arises as a technique to condense a large dataset into a smaller synthetic one that exhibits a similar result in the training process. While image data presents an extensive literature of distillation methods, text dataset distillation has fewer works in comparison. Text dataset distillation initially grew as an adaptation of efforts from the vision universe, as the particularities of the modality became clear obstacles, it rose into a separate branch of research. Several milestones mark the development of this area, such as the introduction of methods that use transformer models, the generation of discrete synthetic text, and the scaling to decoder-only models with over 1B parameters. Despite major advances in modern approaches, the field remains in a maturing phase, with room for improvement on benchmarking standardization, approaches to overcome the discrete nature of text, handling complex tasks, and providing explicit examples of real-world applications. In this report, we review past and recent advances in dataset distillation for text, highlighting different distillation strategies, key contributions, and general challenges.