Benchmarking Generative Latent Variable Models for Speech
Not on the map. Why?
Abstract (arXiv)
Stochastic latent variable models (LVMs) achieve state-of-the-art performance on natural image generation but are still inferior to deterministic models on speech. In this paper, we develop a speech benchmark of popular temporal LVMs and compare them against state-of-the-art deterministic models. We report the likelihood, which is a much used metric in the image domain, but rarely, or incomparably, reported for speech models. To assess the quality of the learned representations, we also compare their usefulness for phoneme recognition. Finally, we adapt the Clockwork VAE, a state-of-the-art temporal LVM for video generation, to the speech domain. Despite being autoregressive only in latent space, we find that the Clockwork VAE can outperform previous LVMs and reduce the gap to deterministic models by using a hierarchy of latent variables.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 22 Feb 2022.
Builds on
38 of 52 references.
References · 38
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2017Neural Discrete Representation Learningcited by 4,650
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2015Variational Inference with Normalizing Flowscited by 2,161
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2021Variational Diffusion Modelscited by 1,145
- 2014NICE: Non-linear Independent Components Estimationcited by 1,052
- 2016Pixel Recurrent Neural Networkscited by 814
- 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
26 more
- 2015Generating Sentences from a Continuous Spacecited by 607
- 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
- 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
- 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
- 2016Ladder Variational Autoencoderscited by 315
- 2015A note on the evaluation of generative modelscited by 291
- 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
- 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
- 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
- 2016Sequential Neural Models with Stochastic Layerscited by 115
- 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
- 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
- 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
- 2020S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generationcited by 63
- 2014Learning Stochastic Recurrent Networkscited by 60
- 2021Hierarchical VAEs Know What They Don't Knowcited by 58
- 2017Z-Forcing: Training Stochastic Recurrent Networkscited by 51
- 2021Consistency Regularization for Variational Auto-Encoderscited by 50
- 2021Clockwork Variational Autoencoderscited by 40
- 2019STCN: Stochastic Temporal Convolutional Networkscited by 26
- 2018Leveraging the Exact Likelihood of Deep Latent Variable Modelscited by 23
- 2021Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Modelscited by 14
- 2018Stochastic WaveNet: A Generative Latent Variable Model for Sequential Datacited by 4
- 2019Re-examination of the Role of Latent Variables in Sequence Modelingcited by 2
Built on by
Citing papers per year: 2023: 1.
1 paper.