# Training language GANs from Scratch

arXiv: https://arxiv.org/abs/1905.09922 · HTML: https://mlsanity.com/paper/1905.09922/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2019.

## Built on by (most cited first, 30 of 31)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [RADAR: Robust AI-Text Detection via Adversarial Learning](https://mlsanity.com/paper/2307.03838/) (2023)
- [Self-conditioned Embedding Diffusion for Text Generation](https://mlsanity.com/paper/2211.04236/) (2022)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion](https://mlsanity.com/paper/2504.16431/) (2025)
- [Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning](https://mlsanity.com/paper/2404.19409/) (2024)
- [Data Redaction from Pre-trained GANs](https://mlsanity.com/paper/2206.14389/) (2022)
- [To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs](https://mlsanity.com/paper/2106.06363/) (2021)
- [Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules](https://mlsanity.com/paper/2210.06184/) (2022)
- [Generative Cooperative Networks for Natural Language Generation](https://mlsanity.com/paper/2201.12320/) (2022)
- [Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMs](https://mlsanity.com/paper/2304.08968/) (2023)
- [Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows](https://mlsanity.com/paper/2507.00425/) (2025)
- [Efficient Reinforcement Learning for Unsupervised Controlled Text Generation](https://mlsanity.com/paper/2204.07696/) (2022)
- [Improved Training of Mixture-of-Experts Language GANs](https://mlsanity.com/paper/2302.11875/) (2023)
- [Regularizing Transformers With Deep Probabilistic Layers](https://mlsanity.com/paper/2108.10764/) (2021)
- [In-Context Ability Transfer for Question Decomposition in Complex QA](https://mlsanity.com/paper/2310.18371/) (2023)
- [Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?](https://mlsanity.com/paper/2108.12275/) (2021)
- [Adversarial learning of neural user simulators for dialogue policy optimisation](https://mlsanity.com/paper/2306.00858/) (2023)
- [Controlling Federated Learning for Covertness](https://mlsanity.com/paper/2308.08825/) (2023)
- [Why GANs are overkill for NLP](https://mlsanity.com/paper/2205.09838/) (2022)
- [Customs Import Declaration Datasets](https://mlsanity.com/paper/2208.02484/) (2022)
- [InitialGAN: A Language GAN with Completely Random Initialization](https://mlsanity.com/paper/2208.02531/) (2022)
- [Molecular Generative Adversarial Network with Multi-Property Optimization](https://mlsanity.com/paper/2404.00081/) (2024)
- [GANetic Loss for Generative Adversarial Networks with a Focus on Medical Applications](https://mlsanity.com/paper/2406.05023/) (2024)
- [Hybrid Generative AI for De Novo Design of Co-Crystals with Enhanced Tabletability](https://mlsanity.com/paper/2410.17005/) (2024)
- [Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey](https://mlsanity.com/paper/2411.09166/) (2024)
- [Joint Text and Label Generation for Spoken Language Understanding](https://mlsanity.com/paper/2105.05052/) (2021)
- [OptAGAN: Entropy-based finetuning on text VAE-GAN](https://mlsanity.com/paper/2109.00239/) (2021)
- [Learning from Bootstrapping and Stepwise Reinforcement Reward: A Semi-Supervised Framework for Text Style Transfer](https://mlsanity.com/paper/2205.09324/) (2022)
- [Unlocking the Power of GANs in Non-Autoregressive Text Generation](https://mlsanity.com/paper/2305.03977/) (2023)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) enjoy great success at image generation, but have proven difficult to train in the domain of natural language. Challenges with gradient estimation, optimization instability, and mode collapse have lead practitioners to resort to maximum likelihood pre-training, followed by small amounts of adversarial fine-tuning. The benefits of GAN fine-tuning for language generation are unclear, as the resulting models produce comparable or worse samples than traditional language models. We show it is in fact possible to train a language GAN from scratch -- without maximum likelihood pre-training. We combine existing techniques such as large batch sizes, dense rewards and discriminator regularization to stabilize and improve language GANs. The resulting model, ScratchGAN, performs comparably to maximum likelihood training on EMNLP2017 News and WikiText-103 corpora according to quality and diversity metrics.