# VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI

arXiv: https://arxiv.org/abs/2509.13767 · HTML: https://mlsanity.com/paper/2509.13767/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [Fully Convolutional Adaptation Networks for Semantic Segmentation](https://mlsanity.com/paper/1804.08286/) (2018)
- [Object Contour Detection with a Fully Convolutional Encoder-Decoder Network](https://mlsanity.com/paper/1603.04530/) (2016)
- [A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images](https://mlsanity.com/paper/2102.07896/) (2021)
- [Multimodal Segmentation for Vocal Tract Modeling](https://mlsanity.com/paper/2406.15754/) (2024)

## Built on by (most cited first, 2 of 2)

- [SIREM: Speech-Informed MRI Reconstruction with Learned Sampling](https://mlsanity.com/paper/2605.18221/) (2026)
- [Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification](https://mlsanity.com/paper/2608.09767/) (2026)

## Abstract (arXiv)

Accurate segmentation of articulatory structures in real-time MRI (rtMRI) remains challenging, as existing methods rely primarily on visual cues and overlook complementary information from synchronized speech signals. We propose VocSegMRI, a multimodal framework integrating video, audio, and phonological inputs via cross-attention fusion and a contrastive learning objective that improves cross-modal alignment and segmentation precision. Evaluated on USC-75 and further validated via zero-shot transfer on USC-TIMIT, VocSegMRI outperforms unimodal and multimodal baselines, with ablations confirming the contribution of each component.