An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator movements, such as ultrasound tongue imaging (UTI) data. An issue of existing AAI methods is only using the personalized acoustic information to derive the general patterns of tongue motions, and thus the quality of generated UTI data is limited. To address this issue, this paper proposes an audio-textual diffusion model for the UTI data generation task. In this model, the inherent acoustic characteristics of individuals related to the tongue motion details are encoded by using wav2vec 2.0, while the ASR transcriptions related to the universality of tongue motions are encoded by using BERT. UTI data are then generated by using a diffusion module. Experimental results showed that the proposed diffusion model could generate high-quality UTI data with clear tongue contour that is crucial for the linguistic analysis and clinical assessment. The project can be found on the websitefootnotehttps://yangyudong2020.github.io/wav2uti/
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 20 of its 27 references that are in the map. First posted on arXiv 9 Mar 2024.
Builds on
20 of 27 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
- 2022Video Diffusion Modelscited by 2,280
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
- 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
- 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
8 more
- 2023Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesiscited by 32
- 2023Diff-E: Diffusion-based Learning for Decoding Imagined Speech EEGcited by 17
- 2020TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videoscited by 12
- 2019DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imagingcited by 6
- 2022Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning modelscited by 4
- 2022Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networkscited by 4
- 2021Exploiting ultrasound tongue imaging for the automatic detection of speech articulation errorscited by 2
- 2023Speech Reconstruction from Silent Tongue and Lip Articulation By Pseudo Target Generation and Domain Adversarial Trainingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech
- 2025A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI
- 2023Creating Personalized Synthetic Voices from Post-Glossectomy Speech with Guided Diffusion Models
- 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
- 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
- 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
- 2025Language translation, and change of accent for speech-to-speech task using diffusion model
- 2024Diffusion as Sound Propagation: Physics-inspired Model for Ultrasound Image Generation