# Learning robust speech representation with an articulatory-regularized variational autoencoder

arXiv: https://arxiv.org/abs/2104.03204 · HTML: https://mlsanity.com/paper/2104.03204/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder](https://mlsanity.com/paper/1610.04019/) (2016)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [Learning Latent Representations for Speech Generation and Transformation](https://mlsanity.com/paper/1704.04222/) (2017)
- [Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder](https://mlsanity.com/paper/1804.02135/) (2018)
- [Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization](https://mlsanity.com/paper/1811.06713/) (2018)
- [Learning Joint Articulatory-Acoustic Representations with Normalizing Flows](https://mlsanity.com/paper/2005.09463/) (2020)

## Built on by (most cited first, 4 of 4)

- [Coding Speech through Vocal Tract Kinematics](https://mlsanity.com/paper/2406.12998/) (2024)
- [Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAE](https://mlsanity.com/paper/2206.08790/) (2022)
- [Repeat after me: Self-supervised learning of acoustic-to-articulatory mapping by vocal imitation](https://mlsanity.com/paper/2204.02269/) (2022)
- [The Mirrornet : Learning Audio Synthesizer Controls Inspired by Sensorimotor Interaction](https://mlsanity.com/paper/2110.05695/) (2021)

## Abstract (arXiv)

It is increasingly considered that human speech perception and production both rely on articulatory representations. In this paper, we investigate whether this type of representation could improve the performances of a deep generative model (here a variational autoencoder) trained to encode and decode acoustic speech features. First we develop an articulatory model able to associate articulatory parameters describing the jaw, tongue, lips and velum configurations with vocal tract shapes and spectral features. Then we incorporate these articulatory parameters into a variational autoencoder applied on spectral features by using a regularization technique that constraints part of the latent space to follow articulatory trajectories. We show that this articulatory constraint improves model training by decreasing time to convergence and reconstruction loss at convergence, and yields better performance in a speech denoising task.