# Synthesizing audio from tongue motion during speech using tagged MRI via transformer

arXiv: https://arxiv.org/abs/2302.07203 · HTML: https://mlsanity.com/paper/2302.07203/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Builds on (most cited first, 7 of 7)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Vid2speech: Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1701.00495/) (2017)
- [Lip2AudSpec: Speech reconstruction from silent lip movements video](https://mlsanity.com/paper/1710.09798/) (2017)
- [Dual-cycle Constrained Bijective VAE-GAN For Tagged-to-Cine Magnetic Resonance Image Synthesis](https://mlsanity.com/paper/2101.05439/) (2021)
- [A Deep Joint Sparse Non-negative Matrix Factorization Framework for Identifying the Common and Subject-specific Functional Units of Tongue Motion During Speech](https://mlsanity.com/paper/2007.04865/) (2020)

## Built on by (most cited first, 1 of 1)

- [Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI](https://mlsanity.com/paper/2402.06984/) (2024)

## Abstract (arXiv)

Investigating the relationship between internal tissue point motion of the tongue and oropharyngeal muscle deformation measured from tagged MRI and intelligible speech can aid in advancing speech motor control theories and developing novel treatment methods for speech related-disorders. However, elucidating the relationship between these two sources of information is challenging, due in part to the disparity in data structure between spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio waveforms. In this work, we present an efficient encoder-decoder translation network for exploring the predictive information inherent in 4D motion fields via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder is based on 3D convolutional spatial modeling and transformer-based temporal modeling. The extracted features are processed by an asymmetric 2D convolution decoder to generate spectrograms that correspond to 4D motion fields. Furthermore, we incorporate a generative adversarial training approach into our framework to further improve synthesis quality on our generated spectrograms. We experiment on 63 paired motion field sequences and speech waveforms, demonstrating that our framework enables the generation of clear audio waveforms from a sequence of motion fields. Thus, our framework has the potential to improve our understanding of the relationship between these two modalities and inform the development of treatments for speech disorders.