# Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling

arXiv: https://arxiv.org/abs/2105.03641 · HTML: https://mlsanity.com/paper/2105.03641/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Language GANs Falling Short](https://mlsanity.com/paper/1811.02549/) (2018)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [Neural Syntactic Preordering for Controlled Paraphrase Generation](https://mlsanity.com/paper/2005.02013/) (2020)
- [Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech](https://mlsanity.com/paper/1805.12589/) (2018)
- [Syntax-guided Controlled Generation of Paraphrases](https://mlsanity.com/paper/2005.08417/) (2020)
- [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://mlsanity.com/paper/1711.05732/) (2017)
- [Importance of Search and Evaluation Strategies in Neural Dialogue Modeling](https://mlsanity.com/paper/1811.00907/) (2018)
- [Linguistic Input Features Improve Neural Machine Translation](https://mlsanity.com/paper/1606.02892/) (2016)
- [Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss](https://mlsanity.com/paper/1902.09191/) (2019)
- [Negative Training for Neural Dialogue Response Generation](https://mlsanity.com/paper/1903.02134/) (2019)
- [On Accurate Evaluation of GANs for Language Generation](https://mlsanity.com/paper/1806.04936/) (2018)
- [The Role of Syntactic Planning in Compositional Image Captioning](https://mlsanity.com/paper/2101.11911/) (2021)
- [Natural Language Generation by Hierarchical Decoding with Linguistic Patterns](https://mlsanity.com/paper/1808.02747/) (2018)
- [F^2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax](https://mlsanity.com/paper/2009.09417/) (2020)
- [Neural Text Generation with Artificial Negative Examples](https://mlsanity.com/paper/2012.14124/) (2020)

## Built on by (most cited first, 2 of 2)

- [Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood](https://mlsanity.com/paper/2406.19874/) (2024)
- [A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning](https://mlsanity.com/paper/2406.14164/) (2024)

## Abstract (arXiv)

Neural text generation models are likely to suffer from the low-diversity problem. Various decoding strategies and training-based methods have been proposed to promote diversity only by exploiting contextual features, but rarely do they consider incorporating syntactic structure clues. In this work, we propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. In detail, we introduce POS Guided Softmax to explicitly model two posterior probabilities: (i) next-POS, and (ii) next-token from the vocabulary of the target POS. A POS Guided Sampling strategy is further proposed to address the low-diversity problem by enriching the diversity of POS. Extensive experiments and human evaluations show that, compared with existing state-of-the-art methods, our POS Guided Softmax and Sampling (POSG) can generate more diverse text while maintaining comparable quality.