# Learning Joint Articulatory-Acoustic Representations with Normalizing Flows

arXiv: https://arxiv.org/abs/2005.09463 · HTML: https://mlsanity.com/paper/2005.09463/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2020.

## Built on by (most cited first, 2 of 2)

- [Learning robust speech representation with an articulatory-regularized variational autoencoder](https://mlsanity.com/paper/2104.03204/) (2021)
- [The Mirrornet : Learning Audio Synthesizer Controls Inspired by Sensorimotor Interaction](https://mlsanity.com/paper/2110.05695/) (2021)

## Abstract (arXiv)

The articulatory geometric configurations of the vocal tract and the acoustic properties of the resultant speech sound are considered to have a strong causal relationship. This paper aims at finding a joint latent representation between the articulatory and acoustic domain for vowel sounds via invertible neural network models, while simultaneously preserving the respective domain-specific features. Our model utilizes a convolutional autoencoder architecture and normalizing flow-based models to allow both forward and inverse mappings in a semi-supervised manner, between the mid-sagittal vocal tract geometry of a two degrees-of-freedom articulatory synthesizer with 1D acoustic wave model and the Mel-spectrogram representation of the synthesized speech sounds. Our approach achieves satisfactory performance in achieving both articulatory-to-acoustic as well as acoustic-to-articulatory mapping, thereby demonstrating our success in achieving a joint encoding of both the domains.