# Exploring Multilingual Text Data Distillation

arXiv: https://arxiv.org/abs/2308.04982 · HTML: https://mlsanity.com/paper/2308.04982/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)

## Built on by (most cited first, 1 of 1)

- [Synthetic Text Generation for Training Large Language Models via Gradient Matching](https://mlsanity.com/paper/2502.17607/) (2025)

## Abstract (arXiv)

With the rise of deep learning, large datasets and complex models have become common, requiring significant computing power. To address this, data distillation has emerged as a technique to quickly train models with lower memory and time requirements. However, data distillation on text-based datasets hasn't been explored much because of the challenges rising due to its discrete nature. Additionally, existing dataset distillation methods often struggle to generalize to new architectures. In the paper, we propose several data distillation techniques for multilingual text classification datasets using language-model-based learning methods. We conduct experiments to analyze their performance in terms of classification strength, and cross-architecture generalization. Furthermore, we investigate the language-specific fairness of the data summaries generated by these methods. Our approach builds upon existing techniques, enhancing cross-architecture generalization in the text data distillation domain.