# Repeat after me: Self-supervised learning of acoustic-to-articulatory mapping by vocal imitation

arXiv: https://arxiv.org/abs/2204.02269 · HTML: https://mlsanity.com/paper/2204.02269/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 5 of 5)

- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [Learning robust speech representation with an articulatory-regularized variational autoencoder](https://mlsanity.com/paper/2104.03204/) (2021)

## Built on by (most cited first, 3 of 3)

- [Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAE](https://mlsanity.com/paper/2206.08790/) (2022)
- [Simulating Articulatory Trajectories with Phonological Feature Interpolation](https://mlsanity.com/paper/2408.04363/) (2024)
- [Developmental Predictive Coding Model for Early Infancy Mono and Bilingual Vocal Continual Learning](https://mlsanity.com/paper/2412.17456/) (2024)

## Abstract (arXiv)

We propose a computational model of speech production combining a pre-trained neural articulatory synthesizer able to reproduce complex speech stimuli from a limited set of interpretable articulatory parameters, a DNN-based internal forward model predicting the sensory consequences of articulatory commands, and an internal inverse model based on a recurrent neural network recovering articulatory commands from the acoustic speech input. Both forward and inverse models are jointly trained in a self-supervised way from raw acoustic-only speech data from different speakers. The imitation simulations are evaluated objectively and subjectively and display quite encouraging performances.