# Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion

arXiv: https://arxiv.org/abs/2104.02194 · HTML: https://mlsanity.com/paper/2104.02194/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Improving RNN Transducer Based ASR with Auxiliary Tasks](https://mlsanity.com/paper/2011.03109/) (2020)
- [Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2010.13878/) (2020)
- [From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.01493/) (2019)
- [G2G: TTS-Driven Pronunciation Learning for Graphemic Hybrid ASR](https://mlsanity.com/paper/1910.12612/) (2019)
- [Benchmarking LF-MMI, CTC and RNN-T Criteria for Streaming ASR](https://mlsanity.com/paper/2011.04785/) (2020)

## Built on by (most cited first, 30 of 101)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022)
- [Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection](https://mlsanity.com/paper/2201.12806/) (2022)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning](https://mlsanity.com/paper/2311.02248/) (2023)
- [End-to-End Speech Recognition Contextualization with Large Language Models](https://mlsanity.com/paper/2309.10917/) (2023)
- [Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric](https://mlsanity.com/paper/2110.05376/) (2021)
- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Search](https://mlsanity.com/paper/2401.10449/) (2024)
- [CTC-Assisted LLM-Based Contextual ASR](https://mlsanity.com/paper/2411.06437/) (2024)
- [Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognition](https://mlsanity.com/paper/2305.05271/) (2023)
- [Towards Contextual Spelling Correction for Customization of End-to-end Speech Recognition Systems](https://mlsanity.com/paper/2203.00888/) (2022)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model](https://mlsanity.com/paper/2209.01250/) (2022)
- [Improving ASR Contextual Biasing with Guided Attention](https://mlsanity.com/paper/2401.08835/) (2024)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [Efficient Domain Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2302.01496/) (2023)
- [Phoneme-aware Encoding for Prefix-tree-based Contextual ASR](https://mlsanity.com/paper/2312.09582/) (2023)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models](https://mlsanity.com/paper/2212.01992/) (2022)
- [Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbation](https://mlsanity.com/paper/2407.10303/) (2024)
- [Spell my name: keyword boosted speech recognition](https://mlsanity.com/paper/2110.02791/) (2021)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [A Language Agnostic Multilingual Streaming On-Device ASR System](https://mlsanity.com/paper/2208.13916/) (2022)
- [Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation](https://mlsanity.com/paper/2302.11192/) (2023)

## Abstract (arXiv)

How to leverage dynamic contextual information in end-to-end speech recognition has remained an active research area. Previous solutions to this problem were either designed for specialized use cases that did not generalize well to open-domain scenarios, did not scale to large biasing lists, or underperformed on rare long-tail words. We address these limitations by proposing a novel solution that combines shallow fusion, trie-based deep biasing, and neural network language model contextualization. These techniques result in significant 19.5% relative Word Error Rate improvement over existing contextual biasing approaches and 5.4%-9.3% improvement compared to a strong hybrid baseline on both open-domain and constrained contextualization tasks, where the targets consist of mostly rare long-tail words. Our final system remains lightweight and modular, allowing for quick modification without model re-training.