# Multilingual Training and Cross-lingual Adaptation on CTC-based Acoustic Model

arXiv: https://arxiv.org/abs/1711.10025 · HTML: https://mlsanity.com/paper/1711.10025/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2017.

## Built on by (most cited first, 6 of 6)

- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [ArTST: Arabic Text and Speech Transformer](https://mlsanity.com/paper/2310.16621/) (2023)
- [Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings](https://mlsanity.com/paper/2107.05038/) (2021)
- [Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition](https://mlsanity.com/paper/2201.11207/) (2022)
- [Language-universal phonetic encoder for low-resource speech recognition](https://mlsanity.com/paper/2305.11576/) (2023)
- [Effectiveness of text to speech pseudo labels for forced alignment and cross lingual pretrained models for low resource speech recognition](https://mlsanity.com/paper/2203.16823/) (2022)

## Abstract (arXiv)

Multilingual models for Automatic Speech Recognition (ASR) are attractive as they have been shown to benefit from more training data, and better lend themselves to adaptation to under-resourced languages. However, initialisation from monolingual context-dependent models leads to an explosion of context-dependent states. Connectionist Temporal Classification (CTC) is a potential solution to this as it performs well with monophone labels. We investigate multilingual CTC in the context of adaptation and regularisation techniques that have been shown to be beneficial in more conventional contexts. The multilingual model is trained to model a universal International Phonetic Alphabet (IPA)-based phone set using the CTC loss function. Learning Hidden Unit Contribution (LHUC) is investigated to perform language adaptive training. In addition, dropout during cross-lingual adaptation is also studied and tested in order to mitigate the overfitting problem. Experiments show that the performance of the universal phoneme-based CTC system can be improved by applying LHUC and it is extensible to new phonemes during cross-lingual adaptation. Updating all the parameters shows consistent improvement on limited data. Applying dropout during adaptation can further improve the system and achieve competitive performance with Deep Neural Network / Hidden Markov Model (DNN/HMM) systems on limited data.