# On Accurate Evaluation of GANs for Language Generation

arXiv: https://arxiv.org/abs/1806.04936 · HTML: https://mlsanity.com/paper/1806.04936/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2018.

## Built on by (most cited first, 20 of 20)

- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Self-conditioned Embedding Diffusion for Text Generation](https://mlsanity.com/paper/2211.04236/) (2022)
- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs](https://mlsanity.com/paper/2106.06363/) (2021)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Auditing and Generating Synthetic Data with Controllable Trust Trade-offs](https://mlsanity.com/paper/2304.10819/) (2023)
- [Fed-TGAN: Federated Learning Framework for Synthesizing Tabular Data](https://mlsanity.com/paper/2108.07927/) (2021)
- [IntentGPT: Few-shot Intent Discovery with Large Language Models](https://mlsanity.com/paper/2411.10670/) (2024)
- [Distribution Aware Metrics for Conditional Natural Language Generation](https://mlsanity.com/paper/2209.07518/) (2022)
- [In-Context Ability Transfer for Question Decomposition in Complex QA](https://mlsanity.com/paper/2310.18371/) (2023)
- [Exploring Human-Like Thinking in Search Simulations with Large Language Models](https://mlsanity.com/paper/2504.07570/) (2025)
- [Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling](https://mlsanity.com/paper/2105.03641/) (2021)
- [Dependency-based Mixture Language Models](https://mlsanity.com/paper/2203.10256/) (2022)
- [HALO: An Ontology for Representing and Categorizing Hallucinations in Large Language Models](https://mlsanity.com/paper/2312.05209/) (2023)
- [Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding](https://mlsanity.com/paper/2603.03312/) (2026)
- [Joint Text and Label Generation for Spoken Language Understanding](https://mlsanity.com/paper/2105.05052/) (2021)
- [OptAGAN: Entropy-based finetuning on text VAE-GAN](https://mlsanity.com/paper/2109.00239/) (2021)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.