# Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter

arXiv: https://arxiv.org/abs/2011.08469 · HTML: https://mlsanity.com/paper/2011.08469/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [Regeneration Learning: A Learning Paradigm for Data Generation](https://mlsanity.com/paper/2301.08846/) (2023)
- [Decoupling recognition and transcription in Mandarin ASR](https://mlsanity.com/paper/2108.01129/) (2021)
- [Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LM](https://mlsanity.com/paper/2209.04062/) (2022)
- [Memory-Efficient Training of RNN-Transducer with Sampled Softmax](https://mlsanity.com/paper/2203.16868/) (2022)
- [On the Effectiveness of Pinyin-Character Dual-Decoding for End-to-End Mandarin Chinese ASR](https://mlsanity.com/paper/2201.10792/) (2022)
- [CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognition](https://mlsanity.com/paper/2511.06860/) (2025)
- [Pronunciation-aware unique character encoding for RNN Transducer-based Mandarin speech recognition](https://mlsanity.com/paper/2207.14578/) (2022)
- [Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition](https://mlsanity.com/paper/2205.11998/) (2022)
- [Transducers with Pronunciation-aware Embeddings for Automatic Speech Recognition](https://mlsanity.com/paper/2404.04295/) (2024)
- [T-SANDHI: Tone Sandhi-aware Adaptive Network with Decoupled Hybrid Injection for Low-resource Taiwanese Hokkien Speech Recognition](https://mlsanity.com/paper/2609.18194/) (2026)

## Abstract (arXiv)

End-to-end models are favored in automatic speech recognition (ASR) because of its simplified system structure and superior performance. Among these models, recurrent neural network transducer (RNN-T) has achieved significant progress in streaming on-device speech recognition because of its high-accuracy and low-latency. RNN-T adopts a prediction network to enhance language information, but its language modeling ability is limited because it still needs paired speech-text data to train. Further strengthening the language modeling ability through extra text data, such as shallow fusion with an external language model, only brings a small performance gain. In view of the fact that Mandarin Chinese is a character-based language and each character is pronounced as a tonal syllable, this paper proposes a novel cascade RNN-T approach to improve the language modeling ability of RNN-T. Our approach firstly uses an RNN-T to transform acoustic feature into syllable sequence, and then converts the syllable sequence into character sequence through an RNN-T-based syllable-to-character converter. Thus a rich text repository can be easily used to strengthen the language model ability. By introducing several important tricks, the cascade RNN-T approach surpasses the character-based RNN-T by a large margin on several Mandarin test sets, with much higher recognition quality and similar latency.