# Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation arXiv: https://arxiv.org/abs/2507.06249 ยท HTML: https://mlsanity.com/paper/2507.06249/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 55 references that are in the map. - First posted on arXiv 4 Jul 2025. ## Builds on (most cited first, 30 of 32) - [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019) - [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021) - [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020) - [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021) - [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023) - [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020) - [Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2002.07510/) (2020) - [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020) - [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021) - [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014) - [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021) - [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022) - [A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning](https://mlsanity.com/paper/2009.08115/) (2020) - [Latent Execution for Neural Program Synthesis](https://mlsanity.com/paper/2107.00101/) (2021) - [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025) - [Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision](https://mlsanity.com/paper/2406.02166/) (2024) - [Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities](https://mlsanity.com/paper/2211.05756/) (2022) - [Massively Multilingual Neural Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1708.01464/) (2017) - [Multilingual Graphemic Hybrid ASR with Massive Data Augmentation](https://mlsanity.com/paper/1909.06522/) (2019) - [CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency](https://mlsanity.com/paper/2005.13326/) (2020) - [Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings](https://mlsanity.com/paper/2107.05038/) (2021) - [Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR](https://mlsanity.com/paper/2111.06799/) (2021) - [Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models](https://mlsanity.com/paper/2005.14001/) (2020) - [Knowledge-Retrieval Task-Oriented Dialog Systems with Semi-Supervision](https://mlsanity.com/paper/2305.13199/) (2023) - [Joint Stochastic Approximation learning of Helmholtz Machines](https://mlsanity.com/paper/1603.06170/) (2016) - [Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation](https://mlsanity.com/paper/2312.03312/) (2023) - [TranUSR: Phoneme-to-word Transcoder Based Unified Speech Representation Learning for Cross-lingual Speech Recognition](https://mlsanity.com/paper/2305.13629/) (2023) ## Built on by (most cited first, 3 of 3) - [CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment](https://mlsanity.com/paper/2602.19574/) (2026) - [Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition](https://mlsanity.com/paper/2603.29217/) (2026) - [Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages](https://mlsanity.com/paper/2608.01281/) (2026) ## Abstract (arXiv) Recently, pre-trained models with phonetic supervision have demonstrated their advantages for crosslingual speech recognition in data efficiency and information sharing across languages. However, a limitation is that a pronunciation lexicon is needed for such phoneme-based crosslingual speech recognition. In this study, we aim to eliminate the need for pronunciation lexicons and propose a latent variable model based method, with phonemes being treated as discrete latent variables. The new method consists of a speech-to-phoneme (S2P) model and a phoneme-to-grapheme (P2G) model, and a grapheme-to-phoneme (G2P) model is introduced as an auxiliary inference model. To jointly train the three models, we utilize the joint stochastic approximation (JSA) algorithm, which is a stochastic extension of the EM (expectation-maximization) algorithm and has demonstrated superior performance particularly in estimating discrete latent variable models. Furthermore, we propose marginal likelihood scoring (MLS) decoding to align inference with the training objective and P2G augmentation to improve the robustness of P2G mapping. Based on the Whistle multilingual pre-trained S2P model, crosslingual experiments are conducted in Polish (130 h) and Indonesian (20 h). With only 10 minutes of phoneme supervision, the new method, JSA-SPG, achieves 5% error rate reductions compared to the best crosslingual fine-tuning approach using subword or full phoneme supervision. Furthermore, it is found that in language domain adaptation (i.e., utilizing cross-domain text-only data), JSA-SPG outperforms the standard practice of language model fusion via the auxiliary support of the G2P model by 9% error rate reductions. To facilitate reproducibility and encourage further exploration in this field, we open-source the JSA-SPG training code and complete pipeline.