# A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images

arXiv: https://arxiv.org/abs/2102.07896 · HTML: https://mlsanity.com/paper/2102.07896/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2021.

## Built on by (most cited first, 21 of 21)

- [Bora: Biomedical Generalist Video Generation Model](https://mlsanity.com/paper/2407.08944/) (2024)
- [Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech](https://mlsanity.com/paper/2409.15525/) (2024)
- [Speech Understanding on Tiny Devices with A Learning Cache](https://mlsanity.com/paper/2311.18188/) (2023)
- [Multimodal Segmentation for Vocal Tract Modeling](https://mlsanity.com/paper/2406.15754/) (2024)
- [Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition](https://mlsanity.com/paper/2505.24059/) (2025)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI](https://mlsanity.com/paper/2503.12102/) (2025)
- [MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows](https://mlsanity.com/paper/2506.01119/) (2025)
- [VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI](https://mlsanity.com/paper/2509.13767/) (2025)
- [An Approach to Simultaneous Acquisition of Real-Time MRI Video, EEG, and Surface EMG for Articulatory, Brain, and Muscle Activity During Speech Production](https://mlsanity.com/paper/2603.04840/) (2026)
- [ARTI-6: Towards Six-dimensional Articulatory Speech Encoding](https://mlsanity.com/paper/2509.21447/) (2025)
- [Speech Codec Probing from Semantic and Phonetic Perspectives](https://mlsanity.com/paper/2603.10371/) (2026)
- [SIREM: Speech-Informed MRI Reconstruction with Learned Sampling](https://mlsanity.com/paper/2605.18221/) (2026)
- [Lightweight, error-tolerant edge detection using memristor-enabled stochastic logics](https://mlsanity.com/paper/2402.16908/) (2024)
- [Open-Source Manually Annotated Vocal Tract Database for Automatic Segmentation from 3D MRI Using Deep Learning: Benchmarking 2D and 3D Convolutional and Transformer Networks](https://mlsanity.com/paper/2501.06229/) (2025)
- [Role of the Pretraining and the Adaptation data sizes for low-resource real-time MRI video segmentation](https://mlsanity.com/paper/2502.14418/) (2025)
- [Speech Audio Generation from dynamic MRI via a Knowledge Enhanced Conditional Variational Autoencoder](https://mlsanity.com/paper/2503.06588/) (2025)
- [A New k-Space Model for Non-Cartesian Fourier Imaging](https://mlsanity.com/paper/2505.05647/) (2025)
- [Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification](https://mlsanity.com/paper/2608.09767/) (2026)
- [Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis](https://mlsanity.com/paper/2609.09757/) (2026)
- [Reconstructing the Vocal Tract with Differentiable Acoustic Simulation](https://mlsanity.com/paper/2609.36737/) (2026)

## Abstract (arXiv)

Real-time magnetic resonance imaging (RT-MRI) of human speech production is enabling significant advances in speech science, linguistics, bio-inspired speech technology development, and clinical applications. Easy access to RT-MRI is however limited, and comprehensive datasets with broad access are needed to catalyze research across numerous domains. The imaging of the rapidly moving articulators and dynamic airway shaping during speech demands high spatio-temporal resolution and robust reconstruction methods. Further, while reconstructed images have been published, to-date there is no open dataset providing raw multi-coil RT-MRI data from an optimized speech production experimental setup. Such datasets could enable new and improved methods for dynamic image reconstruction, artifact correction, feature extraction, and direct extraction of linguistically-relevant biomarkers. The present dataset offers a unique corpus of 2D sagittal-view RT-MRI videos along with synchronized audio for 75 subjects performing linguistically motivated speech tasks, alongside the corresponding first-ever public domain raw RT-MRI data. The dataset also includes 3D volumetric vocal tract MRI during sustained speech sounds and high-resolution static anatomical T2-weighted upper airway MRI for each subject.