Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation
Not on the map. Why?
Abstract (arXiv)
The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Nov 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 2; 2022: 5; 2023: 2.
9 papers.
- 2022You Are What You Write: Preserving Privacy in the Era of Large Language Modelscited by 12
- 2021Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformerscited by 8
- 2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Informationcited by 6
- 2021Affective Decoding for Empathetic Response Generationcited by 4
- 2022Learning to Drop Out: An Adversarial Approach to Training Sequence VAEscited by 3
- 2022Improving Variational Autoencoders with Density Gap-based Regularizationcited by 3
- 2022Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generationcited by 1
- 2022PUF-Phenotype: A Robust and Noise-Resilient Approach to Aid Intra-Group-based Authentication with DRAM-PUFs Using Machine Learningcited by 0
- 2023Requirement Formalisation using Natural Language Processing and Machine Learning: A Systematic Reviewcited by 0
- 2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information26 May 2023
- 2023Requirement Formalisation using Natural Language Processing and Machine Learning: A Systematic Review18 Mar 2023
- 2022Improving Variational Autoencoders with Density Gap-based Regularization1 Nov 2022
- 2022Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation22 Oct 2022
- 2022Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs26 Sep 2022
- 2022PUF-Phenotype: A Robust and Noise-Resilient Approach to Aid Intra-Group-based Authentication with DRAM-PUFs Using Machine Learning11 Jul 2022
- 2022You Are What You Write: Preserving Privacy in the Era of Large Language Models20 Apr 2022
- 2021Affective Decoding for Empathetic Response Generation18 Aug 2021
- 2021Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers30 Mar 2021