# Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech

arXiv: https://arxiv.org/abs/2407.03132 · HTML: https://mlsanity.com/paper/2407.03132/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and Detection](https://mlsanity.com/paper/2312.12810/) (2023)
- [Speaker-Independent Acoustic-to-Articulatory Speech Inversion](https://mlsanity.com/paper/2302.06774/) (2023)
- [Common Phone: A Multilingual Dataset for Robust Acoustic Modelling](https://mlsanity.com/paper/2201.05912/) (2022)
- [The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversion](https://mlsanity.com/paper/2210.16450/) (2022)
- [Estimating articulatory movements in speech production with transformer networks](https://mlsanity.com/paper/2104.05017/) (2021)
- [Combining Frame-Synchronous and Label-Synchronous Systems for Speech Recognition](https://mlsanity.com/paper/2107.00764/) (2021)
- [Acoustic-to-articulatory Inversion based on Speech Decomposition and Auxiliary Feature](https://mlsanity.com/paper/2204.00873/) (2022)
- [Articulation GAN: Unsupervised modeling of articulatory learning](https://mlsanity.com/paper/2210.15173/) (2022)
- [Independent and automatic evaluation of acoustic-to-articulatory inversion models](https://mlsanity.com/paper/1911.06573/) (2019)
- [Acoustic-to-articulatory Speech Inversion with Multi-task Learning](https://mlsanity.com/paper/2205.13755/) (2022)
- [A comparative study of estimating articulatory movements from phoneme sequences and acoustic features](https://mlsanity.com/paper/1910.14375/) (2019)
- [Acoustic-to-Articulatory Speech Inversion Features for Mispronunciation Detection of /r/ in Child Speech Sound Disorders](https://mlsanity.com/paper/2305.16085/) (2023)

## Abstract (arXiv)

This paper introduces a novel combination of two tasks, previously treated separately: acoustic-to-articulatory speech inversion (AAI) and phoneme-to-articulatory (PTA) motion estimation. We refer to this joint task as acoustic phoneme-to-articulatory speech inversion (APTAI) and explore two different approaches, both working speaker- and text-independently during inference. We use a multi-task learning setup, with the end-to-end goal of taking raw speech as input and estimating the corresponding articulatory movements, phoneme sequence, and phoneme alignment. While both proposed approaches share these same requirements, they differ in their way of achieving phoneme-related predictions: one is based on frame classification, the other on a two-staged training procedure and forced alignment. We reach competitive performance of 0.73 mean correlation for the AAI task and achieve up to approximately 87% frame overlap compared to a state-of-the-art text-dependent phoneme force aligner.