# SIREM: Speech-Informed MRI Reconstruction with Learned Sampling

arXiv: https://arxiv.org/abs/2605.18221 · HTML: https://mlsanity.com/paper/2605.18221/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 13 of 13)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [Learning a Variational Network for Reconstruction of Accelerated MRI Data](https://mlsanity.com/paper/1704.00447/) (2017)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [Reducing Uncertainty in Undersampled MRI Reconstruction with Active Acquisition](https://mlsanity.com/paper/1902.03051/) (2019)
- [Learning-based Optimization of the Under-sampling Pattern in MRI](https://mlsanity.com/paper/1901.01960/) (2019)
- [A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images](https://mlsanity.com/paper/2102.07896/) (2021)
- [Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks](https://mlsanity.com/paper/2210.16881/) (2022)
- [Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech](https://mlsanity.com/paper/2409.15525/) (2024)
- [Multimodal Segmentation for Vocal Tract Modeling](https://mlsanity.com/paper/2406.15754/) (2024)
- [A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI](https://mlsanity.com/paper/2503.12102/) (2025)
- [VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI](https://mlsanity.com/paper/2509.13767/) (2025)

## Built on by (most cited first, 1 of 1)

- [Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification](https://mlsanity.com/paper/2608.09767/) (2026)

## Abstract (arXiv)

Real-time magnetic resonance imaging (rtMRI) of speech production enables non-invasive visualization of dynamic vocal-tract motion and is valuable for speech science and clinical assessment. However, rtMRI is fundamentally constrained by trade-offs among spatial resolution, temporal resolution, and acquisition speed, often leading to undersampled k-space measurements and degraded reconstructions. We propose SIREM, a speech-informed MRI reconstruction framework that uses synchronized speech as a cross-modal prior. The central idea is that vocal-tract configurations during speech are correlated with the produced acoustics, making part of the image content predictable from audio. SIREM models each frame as a fusion of an audio-driven component and an MRI-driven component through a spatial weighting map. The audio branch predicts articulator-related structure from speech, while the MRI branch reconstructs complementary content from measured k-space data. We further introduce a learnable soft weighting profile over spiral arms, enabling a differentiable study of how k-space arm usage interacts with speech-informed fusion. This yields a unified multimodal formulation that combines audio-driven prediction, MRI reconstruction, and sampling adaptation. We evaluate SIREM on the USC speech rtMRI benchmark against standard baselines, including gridding, wavelet-based compressed sensing, and total variation. SIREM introduces a speech-informed reconstruction paradigm that operates in a substantially higher-throughput regime than iterative methods while preserving anatomically plausible vocal-tract structure. These results establish an initial benchmark for multimodal speech-informed rtMRI reconstruction and highlight the potential of synchronized speech as an auxiliary prior for fast reconstruction. The source code is available at https://github.com/mdhasanai/SIREM