mlsanityData to 14 Oct 2026

Advancing NAM-to-Speech Conversion with Novel Methods and the MultiNAM Dataset

arXiv 2412.18839 ↗First posted 25 Dec 2024cs.SD cs.AI eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Current Non-Audible Murmur (NAM)-to-speech techniques rely on voice cloning to simulate ground-truth speech from paired whispers. However, the simulated speech often lacks intelligibility and fails to generalize well across different speakers. To address this issue, we focus on learning phoneme-level alignments from paired whispers and text and employ a Text-to-Speech (TTS) system to simulate the ground-truth. To reduce dependence on whispers, we learn phoneme alignments directly from NAMs, though the quality is constrained by the available training data. To further mitigate reliance on NAM/whisper data for ground-truth simulation, we propose incorporating the lip modality to infer speech and introduce a novel diffusion-based method that leverages recent advancements in lip-to-speech technology. Additionally, we release the MultiNAM dataset with over 7.96 hours of paired NAM, whisper, video, and text data from two speakers and benchmark all methods on this dataset. Speech samples and the dataset are available at https://diff-nam.github.io/DiffNAM/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 33 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

15 of 33 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  9. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  10. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2023LipVoicer: Generating Speech from Silent Videos Guided by Lip Readingcited by 22
3 more
  1. 2024Towards Accurate Lip-to-Speech Synthesis in-the-Wildcited by 7
  2. 2023ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representationscited by 4
  3. 2024Towards Improving NAM-to-Speech Synthesis Intelligibility using Self-Supervised Speech Modelscited by 1

Built on by

Not cited yet.