# Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification

arXiv: https://arxiv.org/abs/2608.09767 · HTML: https://mlsanity.com/paper/2608.09767/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration](https://mlsanity.com/paper/2306.06232/) (2023)
- [A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images](https://mlsanity.com/paper/2102.07896/) (2021)
- [Towards Automatic Speech Identification from Vocal Tract Shape Dynamics in Real-time MRI](https://mlsanity.com/paper/1807.11089/) (2018)
- [Speaker dependent articulatory-to-acoustic mapping using real-time MRI of the vocal tract](https://mlsanity.com/paper/2008.00889/) (2020)
- [Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition](https://mlsanity.com/paper/2509.15689/) (2025)
- [Reconstruction of the Complete Vocal Tract Contour Through Acoustic to Articulatory Inversion Using Real-Time MRI Data](https://mlsanity.com/paper/2510.00914/) (2025)
- [Silent Speech and Emotion Recognition from Vocal Tract Shape Dynamics in Real-Time MRI](https://mlsanity.com/paper/2106.08706/) (2021)
- [MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI](https://mlsanity.com/paper/2412.18836/) (2024)
- [A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI](https://mlsanity.com/paper/2503.12102/) (2025)
- [VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI](https://mlsanity.com/paper/2509.13767/) (2025)
- [Multilingual Phonological Feature Recognition with Self-Supervised Speech Models](https://mlsanity.com/paper/2605.25596/) (2026)
- [SIREM: Speech-Informed MRI Reconstruction with Learned Sampling](https://mlsanity.com/paper/2605.18221/) (2026)

## Abstract (arXiv)

Real-time MRI makes it possible to observe vocal-tract articulation during speech, but mapping these articulatory patterns to phonetic and phonological categories remains challenging. We investigate whether PhonoQ, an audio-based model trained to recognize structured phonological features, provides useful information for audio--articulatory modeling. Specifically, we extract representations from PhonoQ's Conformer module, whose training is shaped by supervision for manner, place, voicing, and vowel features. Using articulatory contours with synchronized audio-derived features, we compare WavLM-large and HuBERT-large baselines with models that incorporate PhonoQ-derived representations. Across unseen-speech and unseen-subject settings, these features improve macro-F1 for phonological targets including manner, place, voicing, vowel height, and vowel backness, and also improve fine-grained 39-phoneme classification. In a contour-only inference setting, audio-derived teacher supervision yields modest but consistent gains over contour-only training, indicating that phonological information from synchronized audio can be partially transferred to articulatory models. Finally, posterior analyses show interpretable surface-sensitive patterns consistent with flapping-like /t/ realizations, /t/-/r/ retraction or affrication, and nasal place assimilation.