# Transducers with Pronunciation-aware Embeddings for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2404.04295 · HTML: https://mlsanity.com/paper/2404.04295/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [THCHS-30 : A Free Chinese Speech Corpus](https://mlsanity.com/paper/1512.01882/) (2015)
- [Espresso: A Fast End-to-end Neural Speech Recognition Toolkit](https://mlsanity.com/paper/1909.08723/) (2019)
- [Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter](https://mlsanity.com/paper/2011.08469/) (2020)
- [Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling](https://mlsanity.com/paper/1811.04284/) (2018)
- [PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers](https://mlsanity.com/paper/2303.17131/) (2023)
- [Decoupling recognition and transcription in Mandarin ASR](https://mlsanity.com/paper/2108.01129/) (2021)
- [Pronunciation-aware unique character encoding for RNN Transducer-based Mandarin speech recognition](https://mlsanity.com/paper/2207.14578/) (2022)

## Abstract (arXiv)

This paper proposes Transducers with Pronunciation-aware Embeddings (PET). Unlike conventional Transducers where the decoder embeddings for different tokens are trained independently, the PET model's decoder embedding incorporates shared components for text tokens with the same or similar pronunciations. With experiments conducted in multiple datasets in Mandarin Chinese and Korean, we show that PET models consistently improve speech recognition accuracy compared to conventional Transducers. Our investigation also uncovers a phenomenon that we call error chain reactions. Instead of recognition errors being evenly spread throughout an utterance, they tend to group together, with subsequent errors often following earlier ones. Our analysis shows that PET models effectively mitigate this issue by substantially reducing the likelihood of the model generating additional errors following a prior one. Our implementation will be open-sourced with the NeMo toolkit.