# DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging

arXiv: https://arxiv.org/abs/1904.06083 · HTML: https://mlsanity.com/paper/1904.06083/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 6 of 6)

- [Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition](https://mlsanity.com/paper/2206.07327/) (2022)
- [Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanism](https://mlsanity.com/paper/2106.11769/) (2021)
- [Convolutional Neural Network-Based Age Estimation Using B-Mode Ultrasound Tongue Image](https://mlsanity.com/paper/2101.11245/) (2021)
- [Automatic audiovisual synchronisation for ultrasound tongue imaging](https://mlsanity.com/paper/2105.15162/) (2021)
- [An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data](https://mlsanity.com/paper/2403.05820/) (2024)
- [Acoustic to Articulatory Inversion of Speech; Data Driven Approaches, Challenges, Applications, and Future Scope](https://mlsanity.com/paper/2504.13308/) (2025)

## Abstract (arXiv)

Speech sounds are produced as the coordinated movement of the speaking organs. There are several available methods to model the relation of articulatory movements and the resulting speech signal. The reverse problem is often called as acoustic-to-articulatory inversion (AAI). In this paper we have implemented several different Deep Neural Networks (DNNs) to estimate the articulatory information from the acoustic signal. There are several previous works related to performing this task, but most of them are using ElectroMagnetic Articulography (EMA) for tracking the articulatory movement. Compared to EMA, Ultrasound Tongue Imaging (UTI) is a technique of higher cost-benefit if we take into account equipment cost, portability, safety and visualized structures. Seeing that, our goal is to train a DNN to obtain UT images, when using speech as input. We also test two approaches to represent the articulatory information: 1) the EigenTongue space and 2) the raw ultrasound image. As an objective quality measure for the reconstructed UT images, we use MSE, Structural Similarity Index (SSIM) and Complex-Wavelet SSIM (CW-SSIM). Our experimental results show that CW-SSIM is the most useful error measure in the UTI context. We tested three different system configurations: a) simple DNN composed of 2 hidden layers with 64x64 pixels of an UTI file as target; b) the same simple DNN but with ultrasound images projected to the EigenTongue space as the target; c) and a more complex DNN composed of 5 hidden layers with UTI files projected to the EigenTongue space. In a subjective experiment the subjects found that the neural networks with two hidden layers were more suitable for this inversion task.