# Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis

arXiv: https://arxiv.org/abs/2406.04595 · HTML: https://mlsanity.com/paper/2406.04595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment](https://mlsanity.com/paper/2104.01378/) (2021)
- [Lhotse: a speech data representation library for the modern deep learning ecosystem](https://mlsanity.com/paper/2110.12561/) (2021)
- [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020)
- [Zero-Shot Automatic Pronunciation Assessment](https://mlsanity.com/paper/2305.19563/) (2023)
- [Xiaoicesing 2: A High-Fidelity Singing Voice Synthesizer Based on Generative Adversarial Network](https://mlsanity.com/paper/2210.14666/) (2022)
- [CrossSinger: A Cross-Lingual Multi-Singer High-Fidelity Singing Voice Synthesizer Trained on Monolingual Singers](https://mlsanity.com/paper/2309.12672/) (2023)

## Abstract (arXiv)

Mispronunciation Detection and Diagnosis (MDD) systems, leveraging Automatic Speech Recognition (ASR), face two main challenges in Mandarin Chinese: 1) The two-stage models create an information gap between the phoneme or tone classification stage and the MDD stage. 2) The scarcity of Mandarin MDD datasets limits model training. In this paper, we introduce a stateless RNN-T model for Mandarin MDD, utilizing HuBERT features with pitch embedding through a Pitch Fusion Block. Our model, trained solely on native speaker data, shows a 3% improvement in Phone Error Rate and a 7% increase in False Acceptance Rate over the state-of-the-art baseline in non-native scenarios