# TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks

arXiv: https://arxiv.org/abs/1905.01976 · HTML: https://mlsanity.com/paper/1905.01976/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2019.

## Built on by (most cited first, 14 of 14)

- [Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models](https://mlsanity.com/paper/2408.10189/) (2024)
- [A Review of Text Style Transfer using Deep Learning](https://mlsanity.com/paper/2109.15144/) (2021)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Enhancing the Robustness via Adversarial Learning and Joint Spatial-Temporal Embeddings in Traffic Forecasting](https://mlsanity.com/paper/2208.03063/) (2022)
- [RELIANT: Fair Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2301.01150/) (2023)
- [Regularizing Transformers With Deep Probabilistic Layers](https://mlsanity.com/paper/2108.10764/) (2021)
- [Adversarial Machine Learning in Text Analysis and Generation](https://mlsanity.com/paper/2101.08675/) (2021)
- [Belief Propagation as Diffusion](https://mlsanity.com/paper/2107.12230/) (2021)
- [cGAN-Based High Dimensional IMU Sensor Data Generation for Enhanced Human Activity Recognition in Therapeutic Activities](https://mlsanity.com/paper/2302.07998/) (2023)
- [Training Generative Adversarial Networks with Adaptive Composite Gradient](https://mlsanity.com/paper/2111.05508/) (2021)
- [Survey of Generative Methods for Social Media Analysis](https://mlsanity.com/paper/2112.07041/) (2021)
- [Breaking Down the Task: A Unit-Grained Hybrid Training Framework for Vision and Language Decision Making](https://mlsanity.com/paper/2307.08016/) (2023)
- [Adversarial Conversational Shaping for Intelligent Agents](https://mlsanity.com/paper/2307.11785/) (2023)
- [PAR-AdvGAN: Improving Adversarial Attack Capability with Progressive Auto-Regression AdvGAN](https://mlsanity.com/paper/2502.12207/) (2025)

## Abstract (arXiv)

Text generation is of particular interest in many NLP applications such as machine translation, language modeling, and text summarization. Generative adversarial networks (GANs) achieved a remarkable success in high quality image generation in computer vision,and recently, GANs have gained lots of interest from the NLP community as well. However, achieving similar success in NLP would be more challenging due to the discrete nature of text. In this work, we introduce a method using knowledge distillation to effectively exploit GAN setup for text generation. We demonstrate how autoencoders (AEs) can be used for providing a continuous representation of sentences, which is a smooth representation that assign non-zero probabilities to more than one word. We distill this representation to train the generator to synthesize similar smooth representations. We perform a number of experiments to validate our idea using different datasets and show that our proposed approach yields better performance in terms of the BLEU score and Jensen-Shannon distance (JSD) measure compared to traditional GAN-based text generation approaches without pre-training.