# Language GANs Falling Short

arXiv: https://arxiv.org/abs/1811.02549 · HTML: https://mlsanity.com/paper/1811.02549/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2018.

## Built on by (most cited first, 30 of 97)

- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [Self-Evaluation Guided Beam Search for Reasoning](https://mlsanity.com/paper/2305.00633/) (2023)
- [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://mlsanity.com/paper/2311.01460/) (2023)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Self-conditioned Embedding Diffusion for Text Generation](https://mlsanity.com/paper/2211.04236/) (2022)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [Learning Dynamics of LLM Finetuning](https://mlsanity.com/paper/2407.10490/) (2024)
- [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [Text Generation by Learning from Demonstrations](https://mlsanity.com/paper/2009.07839/) (2020)
- [The Next Chapter: A Study of Large Language Models in Storytelling](https://mlsanity.com/paper/2301.09790/) (2023)
- [Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2211.07634/) (2022)
- [What Context Features Can Transformer Language Models Use?](https://mlsanity.com/paper/2106.08367/) (2021)
- [Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models](https://mlsanity.com/paper/2305.09955/) (2023)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Benchmarking Linguistic Diversity of Large Language Models](https://mlsanity.com/paper/2412.10271/) (2024)
- [Efficient (Soft) Q-Learning for Text Generation with Limited Good Data](https://mlsanity.com/paper/2106.07704/) (2021)
- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [The Homogenizing Effect of Large Language Models on Human Expression and Thought](https://mlsanity.com/paper/2508.01491/) (2025)
- [PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding](https://mlsanity.com/paper/2109.13582/) (2021)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [Toward Joint Language Modeling for Speech Units and Text](https://mlsanity.com/paper/2310.08715/) (2023)
- [Tailoring Language Generation Models under Total Variation Distance](https://mlsanity.com/paper/2302.13344/) (2023)
- [Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning](https://mlsanity.com/paper/2405.15624/) (2024)
- [Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks](https://mlsanity.com/paper/2402.11638/) (2024)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [Coarse-to-Fine: a Hierarchical Diffusion Model for Molecule Generation in 3D](https://mlsanity.com/paper/2305.13266/) (2023)

## Abstract (arXiv)

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort