# Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translator

arXiv: https://arxiv.org/abs/2206.02284 · HTML: https://mlsanity.com/paper/2206.02284/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 5 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Vid2speech: Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1701.00495/) (2017)
- [Lip2AudSpec: Speech reconstruction from silent lip movements video](https://mlsanity.com/paper/1710.09798/) (2017)
- [Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models](https://mlsanity.com/paper/1911.07421/) (2019)
- [Generative Self-training for Cross-domain Unsupervised Tagged-to-Cine MRI Synthesis](https://mlsanity.com/paper/2106.12499/) (2021)
- [Domain Generalization under Conditional and Label Shifts via Variational Bayesian Inference](https://mlsanity.com/paper/2107.10931/) (2021)
- [AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation](https://mlsanity.com/paper/2007.06620/) (2020)
- [Dual-cycle Constrained Bijective VAE-GAN For Tagged-to-Cine Magnetic Resonance Image Synthesis](https://mlsanity.com/paper/2101.05439/) (2021)
- [Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology](https://mlsanity.com/paper/2101.06775/) (2021)
- [Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders](https://mlsanity.com/paper/2104.11598/) (2021)
- [Structure-aware Unsupervised Tagged-to-Cine MRI Synthesis with Self Disentanglement](https://mlsanity.com/paper/2202.12474/) (2022)
- [Deep 3D-CNN for Depression Diagnosis with Facial Video Recording of Self-Rating Depression Scale Questionnaire](https://mlsanity.com/paper/2107.10712/) (2021)

## Built on by (most cited first, 1 of 1)

- [Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI](https://mlsanity.com/paper/2402.06984/) (2024)

## Abstract (arXiv)

Understanding the underlying relationship between tongue and oropharyngeal muscle deformation seen in tagged-MRI and intelligible speech plays an important role in advancing speech motor control theories and treatment of speech related-disorders. Because of their heterogeneous representations, however, direct mapping between the two modalities -- i.e., two-dimensional (mid-sagittal slice) plus time tagged-MRI sequence and its corresponding one-dimensional waveform -- is not straightforward. Instead, we resort to two-dimensional spectrograms as an intermediate representation, which contains both pitch and resonance, from which to develop an end-to-end deep learning framework to translate from a sequence of tagged-MRI to its corresponding audio waveform with limited dataset size.~Our framework is based on a novel fully convolutional asymmetry translator with guidance of a self residual attention strategy to specifically exploit the moving muscular structures during speech.~In addition, we leverage a pairwise correlation of the samples with the same utterances with a latent space representation disentanglement strategy.~Furthermore, we incorporate an adversarial training approach with generative adversarial networks to offer improved realism on our generated spectrograms.~Our experimental results, carried out with a total of 63 tagged-MRI sequences alongside speech acoustics, showed that our framework enabled the generation of clear audio waveforms from a sequence of tagged-MRI, surpassing competing methods. Thus, our framework provides the great potential to help better understand the relationship between the two modalities.