# UML: A Universal Monolingual Output Layer for Multilingual ASR

arXiv: https://arxiv.org/abs/2302.11186 · HTML: https://mlsanity.com/paper/2302.11186/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Multilingual Speech Recognition With A Single End-To-End Model](https://mlsanity.com/paper/1711.01694/) (2017)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Improved training of end-to-end attention models for speech recognition](https://mlsanity.com/paper/1805.03294/) (2018)
- [On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/1811.00241/) (2018)
- [Massively Multilingual Adversarial Speech Recognition](https://mlsanity.com/paper/1904.02210/) (2019)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Transformer-Transducers for Code-Switched Speech Recognition](https://mlsanity.com/paper/2011.15023/) (2020)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification](https://mlsanity.com/paper/2209.06058/) (2022)
- [Towards Language-Universal End-to-End Speech Recognition](https://mlsanity.com/paper/1711.02207/) (2017)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)

## Built on by (most cited first, 3 of 3)

- [Learn and Don't Forget: Adding a New Language to ASR Foundation Models](https://mlsanity.com/paper/2407.06800/) (2024)
- [Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation](https://mlsanity.com/paper/2502.17380/) (2025)
- [Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training](https://mlsanity.com/paper/2303.00786/) (2023)

## Abstract (arXiv)

Word-piece models (WPMs) are commonly used subword units in state-of-the-art end-to-end automatic speech recognition (ASR) systems. For multilingual ASR, due to the differences in written scripts across languages, multilingual WPMs bring the challenges of having overly large output layers and scaling to more languages. In this work, we propose a universal monolingual output layer (UML) to address such problems. Instead of one output node for only one WPM, UML re-associates each output node with multiple WPMs, one for each language, and results in a smaller monolingual output layer shared across languages. Consequently, the UML enables to switch in the interpretation of each output node depending on the language of the input speech. Experimental results on an 11-language voice search task demonstrated the feasibility of using UML for high-quality and high-efficiency multilingual streaming ASR.