# Bilingual End-to-End ASR with Byte-Level Subwords

arXiv: https://arxiv.org/abs/2205.00485 · HTML: https://mlsanity.com/paper/2205.00485/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 1 May 2022.

## Builds on (most cited first, 12 of 12)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [Multilingual Language Processing From Bytes](https://mlsanity.com/paper/1512.00103/) (2015)
- [Streaming End-to-End Bilingual ASR Systems with Joint Language Identification](https://mlsanity.com/paper/2007.03900/) (2020)
- [A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese](https://mlsanity.com/paper/1805.06239/) (2018)
- [Online Automatic Speech Recognition with Listen, Attend and Spell Model](https://mlsanity.com/paper/2008.05514/) (2020)
- [A comparable study of modeling units for end-to-end Mandarin speech recognition](https://mlsanity.com/paper/1805.03832/) (2018)

## Built on by (most cited first, 4 of 4)

- [Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation](https://mlsanity.com/paper/2210.12214/) (2022)
- [Optimizing Byte-level Representation for End-to-end ASR](https://mlsanity.com/paper/2406.09676/) (2024)
- [Romanization Encoding For Multilingual ASR](https://mlsanity.com/paper/2407.04368/) (2024)
- [BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition](https://mlsanity.com/paper/2602.01717/) (2026)

## Abstract (arXiv)

In this paper, we investigate how the output representation of an end-to-end neural network affects multilingual automatic speech recognition (ASR). We study different representations including character-level, byte-level, byte pair encoding (BPE), and byte-level byte pair encoding (BBPE) representations, and analyze their strengths and weaknesses. We focus on developing a single end-to-end model to support utterance-based bilingual ASR, where speakers do not alternate between two languages in a single utterance but may change languages across utterances. We conduct our experiments on English and Mandarin dictation tasks, and we find that BBPE with penalty schemes can improve utterance-based bilingual ASR performance by 2% to 5% relative even with smaller number of outputs and fewer parameters. We conclude with analysis that indicates directions for further improving multilingual ASR.