# How Phonotactics Affect Multilingual and Zero-shot ASR Performance

arXiv: https://arxiv.org/abs/2010.12104 · HTML: https://mlsanity.com/paper/2010.12104/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 10 of 10)

- [Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings](https://mlsanity.com/paper/2107.05038/) (2021)
- [Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR](https://mlsanity.com/paper/2111.06799/) (2021)
- [Differentiable Allophone Graphs for Language-Universal Speech Recognition](https://mlsanity.com/paper/2107.11628/) (2021)
- [SQuId: Measuring Speech Naturalness in Many Languages](https://mlsanity.com/paper/2210.06324/) (2022)
- [Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition](https://mlsanity.com/paper/2201.11207/) (2022)
- [Investigating the Impact of Cross-lingual Acoustic-Phonetic Similarities on Multilingual Speech Recognition](https://mlsanity.com/paper/2207.03390/) (2022)
- [Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition](https://mlsanity.com/paper/2305.11569/) (2023)
- [Unsupervised Acoustic Unit Discovery by Leveraging a Language-Independent Subword Discriminative Feature Representation](https://mlsanity.com/paper/2104.00994/) (2021)
- [Non-Linear Pairwise Language Mappings for Low-Resource Multilingual Acoustic Model Fusion](https://mlsanity.com/paper/2207.03391/) (2022)
- [Language-universal phonetic encoder for low-resource speech recognition](https://mlsanity.com/paper/2305.11576/) (2023)

## Abstract (arXiv)

The idea of combining multiple languages' recordings to train a single automatic speech recognition (ASR) model brings the promise of the emergence of universal speech representation. Recently, a Transformer encoder-decoder model has been shown to leverage multilingual data well in IPA transcriptions of languages presented during training. However, the representations it learned were not successful in zero-shot transfer to unseen languages. Because that model lacks an explicit factorization of the acoustic model (AM) and language model (LM), it is unclear to what degree the performance suffered from differences in pronunciation or the mismatch in phonotactics. To gain more insight into the factors limiting zero-shot ASR transfer, we replace the encoder-decoder with a hybrid ASR system consisting of a separate AM and LM. Then, we perform an extensive evaluation of monolingual, multilingual, and crosslingual (zero-shot) acoustic and language models on a set of 13 phonetically diverse languages. We show that the gain from modeling crosslingual phonotactics is limited, and imposing a too strong model can hurt the zero-shot transfer. Furthermore, we find that a multilingual LM hurts a multilingual ASR system's performance, and retaining only the target language's phonotactic data in LM training is preferable.