# On the Effectiveness of Pinyin-Character Dual-Decoding for End-to-End Mandarin Chinese ASR

arXiv: https://arxiv.org/abs/2201.10792 · HTML: https://mlsanity.com/paper/2201.10792/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Tied Multitask Learning for Neural Speech Translation](https://mlsanity.com/paper/1802.06655/) (2018)
- [Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation](https://mlsanity.com/paper/2011.00747/) (2020)
- [Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer](https://mlsanity.com/paper/1910.13923/) (2019)
- [Self-Attention Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.13037/) (2019)
- [Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding](https://mlsanity.com/paper/1912.07240/) (2019)
- [Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese](https://mlsanity.com/paper/1804.10752/) (2018)
- [Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter](https://mlsanity.com/paper/2011.08469/) (2020)
- [A comparable study of modeling units for end-to-end Mandarin speech recognition](https://mlsanity.com/paper/1805.03832/) (2018)
- [Research on Modeling Units of Transformer Transducer for Mandarin Speech Recognition](https://mlsanity.com/paper/2004.13522/) (2020)

## Built on by (most cited first, 2 of 2)

- [Romanization Encoding For Multilingual ASR](https://mlsanity.com/paper/2407.04368/) (2024)
- [For the Misgendered Chinese in Gender Bias Research: Multi-Task Learning with Knowledge Distillation for Pinyin Name-Gender Prediction](https://mlsanity.com/paper/2405.06221/) (2024)

## Abstract (arXiv)

End-to-end automatic speech recognition (ASR) has achieved promising results. However, most existing end-to-end ASR methods neglect the use of specific language characteristics. For Mandarin Chinese ASR tasks, there exist mutual promotion relationship between Pinyin and Character where Chinese characters can be romanized by Pinyin. Based on the above intuition, we first investigate types of end-to-end encoder-decoder based models in the single-input dual-output (SIDO) multi-task framework, after which a novel asynchronous decoding with fuzzy Pinyin sampling method is proposed according to the one-to-one correspondence characteristics between Pinyin and Character. Furthermore, we proposed a two-stage training strategy to make training more stable and converge faster. The results on the test sets of AISHELL-1 dataset show that the proposed enhanced dual-decoder model without a language model is improved by a big margin compared to strong baseline models.