# Speech Audio Synthesis from Tagged MRI and Non-Negative Matrix Factorization via Plastic Transformer

arXiv: https://arxiv.org/abs/2309.14586 · HTML: https://mlsanity.com/paper/2309.14586/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [ResT: An Efficient Transformer for Visual Recognition](https://mlsanity.com/paper/2105.13677/) (2021)
- [Vid2speech: Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1701.00495/) (2017)
- [LightViT: Towards Light-Weight Convolution-Free Vision Transformers](https://mlsanity.com/paper/2207.05557/) (2022)
- [Lip2AudSpec: Speech reconstruction from silent lip movements video](https://mlsanity.com/paper/1710.09798/) (2017)
- [Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models](https://mlsanity.com/paper/1911.07421/) (2019)
- [Domain Generalization under Conditional and Label Shifts via Variational Bayesian Inference](https://mlsanity.com/paper/2107.10931/) (2021)
- [A Deep Joint Sparse Non-negative Matrix Factorization Framework for Identifying the Common and Subject-specific Functional Units of Tongue Motion During Speech](https://mlsanity.com/paper/2007.04865/) (2020)
- [A Sparse Non-negative Matrix Factorization Framework for Identifying Functional Units of Tongue Behavior from MRI](https://mlsanity.com/paper/1804.05370/) (2018)

## Built on by (most cited first, 3 of 3)

- [A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI](https://mlsanity.com/paper/2503.12102/) (2025)
- [Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI](https://mlsanity.com/paper/2402.06984/) (2024)
- [Speech Audio Generation from dynamic MRI via a Knowledge Enhanced Conditional Variational Autoencoder](https://mlsanity.com/paper/2503.06588/) (2025)

## Abstract (arXiv)

The tongue's intricate 3D structure, comprising localized functional units, plays a crucial role in the production of speech. When measured using tagged MRI, these functional units exhibit cohesive displacements and derived quantities that facilitate the complex process of speech production. Non-negative matrix factorization-based approaches have been shown to estimate the functional units through motion features, yielding a set of building blocks and a corresponding weighting map. Investigating the link between weighting maps and speech acoustics can offer significant insights into the intricate process of speech production. To this end, in this work, we utilize two-dimensional spectrograms as a proxy representation, and develop an end-to-end deep learning framework for translating weighting maps to their corresponding audio waveforms. Our proposed plastic light transformer (PLT) framework is based on directional product relative position bias and single-level spatial pyramid pooling, thus enabling flexible processing of weighting maps with variable size to fixed-size spectrograms, without input information loss or dimension expansion. Additionally, our PLT framework efficiently models the global correlation of wide matrix input. To improve the realism of our generated spectrograms with relatively limited training samples, we apply pair-wise utterance consistency with Maximum Mean Discrepancy constraint and adversarial training. Experimental results on a dataset of 29 subjects speaking two utterances demonstrated that our framework is able to synthesize speech audio waveforms from weighting maps, outperforming conventional convolution and transformer models.