# Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech

arXiv: https://arxiv.org/abs/2409.15525 · HTML: https://mlsanity.com/paper/2409.15525/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 10 of its 30 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 10 of 10)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos](https://mlsanity.com/paper/2407.16124/) (2024)
- [Speech Driven Video Editing via an Audio-Conditioned Diffusion Model](https://mlsanity.com/paper/2301.04474/) (2023)
- [A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images](https://mlsanity.com/paper/2102.07896/) (2021)
- [Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks](https://mlsanity.com/paper/2210.16881/) (2022)
- [Speaker dependent acoustic-to-articulatory inversion using real-time MRI of the vocal tract](https://mlsanity.com/paper/2008.02098/) (2020)
- [Towards Streaming Speech-to-Avatar Synthesis](https://mlsanity.com/paper/2310.16287/) (2023)
- [CiwaGAN: Articulatory information exchange](https://mlsanity.com/paper/2309.07861/) (2023)

## Built on by (most cited first, 4 of 4)

- [A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI](https://mlsanity.com/paper/2503.12102/) (2025)
- [SIREM: Speech-Informed MRI Reconstruction with Learned Sampling](https://mlsanity.com/paper/2605.18221/) (2026)
- [Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis](https://mlsanity.com/paper/2609.09757/) (2026)
- [Reconstructing the Vocal Tract with Differentiable Acoustic Simulation](https://mlsanity.com/paper/2609.36737/) (2026)

## Abstract (arXiv)

Understanding speech production both visually and kinematically can inform second language learning system designs, as well as the creation of speaking characters in video games and animations. In this work, we introduce a data-driven method to visually represent articulator motion in Magnetic Resonance Imaging (MRI) videos of the human vocal tract during speech based on arbitrary audio or speech input. We leverage large pre-trained speech models, which are embedded with prior knowledge, to generalize the visual domain to unseen data using a speech-to-video diffusion model. Our findings demonstrate that the visual generation significantly benefits from the pre-trained speech representations. We also observed that evaluating phonemes in isolation is challenging but becomes more straightforward when assessed within the context of spoken words. Limitations of the current results include the presence of unsmooth tongue motion and video distortion when the tongue contacts the palate.