# MaskGAN: Better Text Generation via Filling in the______

arXiv: https://arxiv.org/abs/1801.07736 · HTML: https://mlsanity.com/paper/1801.07736/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2018.

## Built on by (most cited first, 30 of 89)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [FENeRF: Face Editing in Neural Radiance Fields](https://mlsanity.com/paper/2111.15490/) (2021)
- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://mlsanity.com/paper/2109.10686/) (2021)
- [Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks](https://mlsanity.com/paper/2108.00298/) (2021)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Deep Imputation of Missing Values in Time Series Health Data: A Review with Benchmarking](https://mlsanity.com/paper/2302.10902/) (2023)
- [Domain Adaptive Fake News Detection via Reinforcement Learning](https://mlsanity.com/paper/2202.08159/) (2022)
- [NRTSI: Non-Recurrent Time Series Imputation](https://mlsanity.com/paper/2102.03340/) (2021)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames](https://mlsanity.com/paper/2102.12841/) (2021)
- [A survey on text generation using generative adversarial networks](https://mlsanity.com/paper/2212.11119/) (2022)
- [TextBox: A Unified, Modularized, and Extensible Framework for Text Generation](https://mlsanity.com/paper/2101.02046/) (2021)
- [Generative Adversarial Networks: A Survey Towards Private and Secure Applications](https://mlsanity.com/paper/2106.03785/) (2021)
- [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://mlsanity.com/paper/2102.01909/) (2021)
- [Barbershop: GAN-based Image Compositing using Segmentation Masks](https://mlsanity.com/paper/2106.01505/) (2021)
- [Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing](https://mlsanity.com/paper/2107.04007/) (2021)
- [Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding](https://mlsanity.com/paper/2108.05064/) (2021)
- [Guiding Visual Question Generation](https://mlsanity.com/paper/2110.08226/) (2021)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [House-GAN++: Generative Adversarial Layout Refinement Networks](https://mlsanity.com/paper/2103.02574/) (2021)

## Abstract (arXiv)

Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of the quality of the generated text. Additionally, these models are typically trained via maxi- mum likelihood and teacher forcing. These methods are well-suited to optimizing perplexity but can result in poor sample quality since generating text requires conditioning on sequences of words that may have never been observed at training time. We propose to improve sample quality using Generative Adversarial Networks (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them. We claim that validation perplexity alone is not indicative of the quality of text generated by a model. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic conditional and unconditional text samples compared to a maximum likelihood trained model.