# Language-universal phonetic encoder for low-resource speech recognition

arXiv: https://arxiv.org/abs/2305.11576 · HTML: https://mlsanity.com/paper/2305.11576/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Multilingual Speech Recognition With A Single End-To-End Model](https://mlsanity.com/paper/1711.01694/) (2017)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [wav2letter++: The Fastest Open-source Speech Recognition System](https://mlsanity.com/paper/1812.07625/) (2018)
- [MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition](https://mlsanity.com/paper/2102.12664/) (2021)
- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Multilingual Language Processing From Bytes](https://mlsanity.com/paper/1512.00103/) (2015)
- [Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages](https://mlsanity.com/paper/1806.05059/) (2018)
- [Towards Language-Universal End-to-End Speech Recognition](https://mlsanity.com/paper/1711.02207/) (2017)
- [XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition](https://mlsanity.com/paper/2103.08207/) (2021)
- [That Sounds Familiar: an Analysis of Phonetic Representations Transfer Across Languages](https://mlsanity.com/paper/2005.08118/) (2020)
- [How Phonotactics Affect Multilingual and Zero-shot ASR Performance](https://mlsanity.com/paper/2010.12104/) (2020)
- [Hierarchical Softmax for End-to-End Low-resource Multilingual Speech Recognition](https://mlsanity.com/paper/2204.03855/) (2022)
- [Multilingual Training and Cross-lingual Adaptation on CTC-based Acoustic Model](https://mlsanity.com/paper/1711.10025/) (2017)
- [Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition](https://mlsanity.com/paper/2201.11207/) (2022)

## Built on by (most cited first, 2 of 2)

- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Scaling Human and G2P Supervision for Robust Phonetic Transcription](https://mlsanity.com/paper/2606.16019/) (2026)

## Abstract (arXiv)

Multilingual training is effective in improving low-resource ASR, which may partially be explained by phonetic representation sharing between languages. In end-to-end (E2E) ASR systems, graphemes are often used as basic modeling units, however graphemes may not be ideal for multilingual phonetic sharing. In this paper, we leverage International Phonetic Alphabet (IPA) based language-universal phonetic model to improve low-resource ASR performances, for the first time within the attention encoder-decoder architecture. We propose an adaptation method on the phonetic IPA model to further improve the proposed approach on extreme low-resource languages. Experiments carried out on the open-source MLS corpus and our internal databases show our approach outperforms baseline monolingual models and most state-of-the-art works. Our main approach and adaptation are effective on extremely low-resource languages, even within domain- and language-mismatched scenarios.