# Non-Linear Pairwise Language Mappings for Low-Resource Multilingual Acoustic Model Fusion

arXiv: https://arxiv.org/abs/2207.03391 · HTML: https://mlsanity.com/paper/2207.03391/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 36 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 7 of 7)

- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018)
- [That Sounds Familiar: an Analysis of Phonetic Representations Transfer Across Languages](https://mlsanity.com/paper/2005.08118/) (2020)
- [How Phonotactics Affect Multilingual and Zero-shot ASR Performance](https://mlsanity.com/paper/2010.12104/) (2020)

## Built on by (most cited first, 2 of 2)

- [Learning Cross-lingual Mappings for Data Augmentation to Improve Low-Resource Speech Recognition](https://mlsanity.com/paper/2306.08577/) (2023)
- [MUST: A Multilingual Student-Teacher Learning approach for low-resource speech recognition](https://mlsanity.com/paper/2310.18865/) (2023)

## Abstract (arXiv)

Multilingual speech recognition has drawn significant attention as an effective way to compensate data scarcity for low-resource languages. End-to-end (e2e) modelling is preferred over conventional hybrid systems, mainly because of no lexicon requirement. However, hybrid DNN-HMMs still outperform e2e models in limited data scenarios. Furthermore, the problem of manual lexicon creation has been alleviated by publicly available trained models of grapheme-to-phoneme (G2P) and text to IPA transliteration for a lot of languages. In this paper, a novel approach of hybrid DNN-HMM acoustic models fusion is proposed in a multilingual setup for the low-resource languages. Posterior distributions from different monolingual acoustic models, against a target language speech signal, are fused together. A separate regression neural network is trained for each source-target language pair to transform posteriors from source acoustic model to the target language. These networks require very limited data as compared to the ASR training. Posterior fusion yields a relative gain of 14.65% and 6.5% when compared with multilingual and monolingual baselines respectively. Cross-lingual model fusion shows that the comparable results can be achieved without using posteriors from the language dependent ASR.