# WakaVT: A Sequential Variational Transformer for Waka Generation

arXiv: https://arxiv.org/abs/2104.00426 · HTML: https://mlsanity.com/paper/2104.00426/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 58 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions](https://mlsanity.com/paper/2011.06801/) (2020)
- [Variational Transformers for Diverse Response Generation](https://mlsanity.com/paper/2003.12738/) (2020)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [Chinese Poetry Generation with Planning based Neural Network](https://mlsanity.com/paper/1610.09889/) (2016)
- [MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space](https://mlsanity.com/paper/2003.06094/) (2020)
- [Generating Thematic Chinese Poetry using Conditional Variational Autoencoders with Hybrid Decoders](https://mlsanity.com/paper/1711.07632/) (2017)
- [Generating Major Types of Chinese Classical Poetry in a Uniformed Framework](https://mlsanity.com/paper/2003.11528/) (2020)
- [Flexible and Creative Chinese Poetry Generation Using Neural Memory](https://mlsanity.com/paper/1705.03773/) (2017)
- [An Iterative Polishing Framework based on Quality Aware Masked Language Model for Chinese Poetry Generation](https://mlsanity.com/paper/1911.13182/) (2019)
- [Compose Like Humans: Jointly Improving the Coherence and Novelty for Modern Chinese Poetry Generation](https://mlsanity.com/paper/2005.01556/) (2020)
- [How Images Inspire Poems: Generating Classical Chinese Poetry from Images with Memory Networks](https://mlsanity.com/paper/1803.02994/) (2018)
- [A Stochastic Decoder for Neural Machine Translation](https://mlsanity.com/paper/1805.10844/) (2018)
- [A Multi-Modal Chinese Poetry Generation Model](https://mlsanity.com/paper/1806.09792/) (2018)

## Abstract (arXiv)

Poetry generation has long been a challenge for artificial intelligence. In the scope of Japanese poetry generation, many researchers have paid attention to Haiku generation, but few have focused on Waka generation. To further explore the creative potential of natural language generation systems in Japanese poetry creation, we propose a novel Waka generation model, WakaVT, which automatically produces Waka poems given user-specified keywords. Firstly, an additive mask-based approach is presented to satisfy the form constraint. Secondly, the structures of Transformer and variational autoencoder are integrated to enhance the quality of generated content. Specifically, to obtain novelty and diversity, WakaVT employs a sequence of latent variables, which effectively captures word-level variability in Waka data. To improve linguistic quality in terms of fluency, coherence, and meaningfulness, we further propose the fused multilevel self-attention mechanism, which properly models the hierarchical linguistic structure of Waka. To the best of our knowledge, we are the first to investigate Waka generation with models based on Transformer and/or variational autoencoder. Both objective and subjective evaluation results demonstrate that our model outperforms baselines significantly.