# An efficient text augmentation approach for contextualized Mandarin speech recognition

arXiv: https://arxiv.org/abs/2406.09950 · HTML: https://mlsanity.com/paper/2406.09950/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [MAESTRO: Matched Speech Text Representations through Modality Matching](https://mlsanity.com/paper/2204.03409/) (2022)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data](https://mlsanity.com/paper/2209.15329/) (2022)
- [Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection](https://mlsanity.com/paper/2201.12806/) (2022)
- [The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders](https://mlsanity.com/paper/2109.06939/) (2021)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2012.09466/) (2020)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [CB-Conformer: Contextual biasing Conformer for biased word recognition](https://mlsanity.com/paper/2304.09607/) (2023)
- [Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation](https://mlsanity.com/paper/2302.11192/) (2023)
- [End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator](https://mlsanity.com/paper/2210.16554/) (2022)
- [Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer](https://mlsanity.com/paper/2311.08966/) (2023)
- [Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer](https://mlsanity.com/paper/2306.04076/) (2023)
- [Spike-Triggered Contextual Biasing for End-to-End Mandarin Speech Recognition](https://mlsanity.com/paper/2310.04657/) (2023)

## Built on by (most cited first, 2 of 2)

- [XCB: an effective contextual biasing approach to bias cross-lingual phrases in speech recognition](https://mlsanity.com/paper/2408.10524/) (2024)
- [M-CIF: Multi-Scale Alignment For CIF-Based Non-Autoregressive ASR](https://mlsanity.com/paper/2510.22172/) (2025)

## Abstract (arXiv)

Although contextualized automatic speech recognition (ASR) systems are commonly used to improve the recognition of uncommon words, their effectiveness is hindered by the inherent limitations of speech-text data availability. To address this challenge, our study proposes to leverage extensive text-only datasets and contextualize pre-trained ASR models using a straightforward text-augmentation (TA) technique, all while keeping computational costs minimal. In particular, to contextualize a pre-trained CIF-based ASR, we construct a codebook using limited speech-text data. By utilizing a simple codebook lookup process, we convert available text-only data into latent text embeddings. These embeddings then enhance the inputs for the contextualized ASR. Our experiments on diverse Mandarin test sets demonstrate that our TA approach significantly boosts recognition performance. The top-performing system shows relative CER improvements of up to 30% on rare words and 15% across all words in general.