# Acquiring Pronunciation Knowledge from Transcribed Speech Audio via Multi-task Learning

arXiv: https://arxiv.org/abs/2409.09891 · HTML: https://mlsanity.com/paper/2409.09891/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Hi-Fi Multi-Speaker English TTS Dataset](https://mlsanity.com/paper/2104.01497/) (2021)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis](https://mlsanity.com/paper/1910.10288/) (2019)
- [Revisiting Over-Smoothness in Text to Speech](https://mlsanity.com/paper/2202.13066/) (2022)
- [Massively Multilingual Neural Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1708.01464/) (2017)
- [SoundChoice: Grapheme-to-Phoneme Models with Semantic Disambiguation](https://mlsanity.com/paper/2207.13703/) (2022)
- [A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis](https://mlsanity.com/paper/1911.04111/) (2019)
- [Scalable Multilingual Frontend for TTS](https://mlsanity.com/paper/2004.04934/) (2020)
- [Improving Arabic Diacritization by Learning to Diacritize and Translate](https://mlsanity.com/paper/2109.14150/) (2021)
- [A unified front-end framework for English text-to-speech synthesis](https://mlsanity.com/paper/2305.10666/) (2023)
- [Towards Robust FastSpeech 2 by Modelling Residual Multimodality](https://mlsanity.com/paper/2306.01442/) (2023)
- [Improving grapheme-to-phoneme conversion by learning pronunciations from speech recordings](https://mlsanity.com/paper/2307.16643/) (2023)
- [Multilingual context-based pronunciation learning for Text-to-Speech](https://mlsanity.com/paper/2307.16709/) (2023)
- [Multi-Task Learning for Front-End Text Processing in TTS](https://mlsanity.com/paper/2401.06321/) (2024)

## Built on by (most cited first, 1 of 1)

- [ReNikud: Audio-Supervised Hebrew Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2606.20179/) (2026)

## Abstract (arXiv)

Recent work has shown the feasibility and benefit of bootstrapping an integrated sequence-to-sequence (Seq2Seq) linguistic frontend from a traditional pipeline-based frontend for text-to-speech (TTS). To overcome the fixed lexical coverage of bootstrapping training data, previous work has proposed to leverage easily accessible transcribed speech audio as an additional training source for acquiring novel pronunciation knowledge for uncovered words, which relies on an auxiliary ASR model as part of a cumbersome implementation flow. In this work, we propose an alternative method to leverage transcribed speech audio as an additional training source, based on multi-task learning (MTL). Experiments show that, compared to a baseline Seq2Seq frontend, the proposed MTL-based method reduces PER from 2.5% to 1.6% for those word types covered exclusively in transcribed speech audio, achieving a similar performance to the previous method but with a much simpler implementation flow.