# TranUSR: Phoneme-to-word Transcoder Based Unified Speech Representation Learning for Cross-lingual Speech Recognition

arXiv: https://arxiv.org/abs/2305.13629 · HTML: https://mlsanity.com/paper/2305.13629/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2023.

## Built on by (most cited first, 4 of 4)

- [Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation](https://mlsanity.com/paper/2312.03312/) (2023)
- [SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition](https://mlsanity.com/paper/2309.16937/) (2023)
- [LLM-based phoneme-to-grapheme for phoneme-based speech recognition](https://mlsanity.com/paper/2506.04711/) (2025)
- [Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation](https://mlsanity.com/paper/2507.06249/) (2025)

## Abstract (arXiv)

UniSpeech has achieved superior performance in cross-lingual automatic speech recognition (ASR) by explicitly aligning latent representations to phoneme units using multi-task self-supervised learning. While the learned representations transfer well from high-resource to low-resource languages, predicting words directly from these phonetic representations in downstream ASR is challenging. In this paper, we propose TranUSR, a two-stage model comprising a pre-trained UniData2vec and a phoneme-to-word Transcoder. Different from UniSpeech, UniData2vec replaces the quantized discrete representations with continuous and contextual representations from a teacher model for phonetically-aware pre-training. Then, Transcoder learns to translate phonemes to words with the aid of extra texts, enabling direct word generation. Experiments on Common Voice show that UniData2vec reduces PER by 5.3% compared to UniSpeech, while Transcoder yields a 14.4% WER reduction compared to grapheme fine-tuning.