# Context-Aware Transformer Transducer for Speech Recognition

arXiv: https://arxiv.org/abs/2111.03250 · HTML: https://mlsanity.com/paper/2111.03250/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 5 Nov 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020)
- [End-to-end contextual speech recognition using class language models and a token passing decoder](https://mlsanity.com/paper/1812.02142/) (2018)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [Self-Attention Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.13037/) (2019)
- [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021)
- [Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model](https://mlsanity.com/paper/2005.07394/) (2020)

## Built on by (most cited first, 30 of 56)

- [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022)
- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognition](https://mlsanity.com/paper/2305.05271/) (2023)
- [Improving ASR Contextual Biasing with Guided Attention](https://mlsanity.com/paper/2401.08835/) (2024)
- [Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducer](https://mlsanity.com/paper/2301.06735/) (2023)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbation](https://mlsanity.com/paper/2407.10303/) (2024)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithm](https://mlsanity.com/paper/2310.00178/) (2023)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)
- [Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition](https://mlsanity.com/paper/2304.08862/) (2023)
- [Automatic Speech Recognition with BERT and CTC Transformers: A Review](https://mlsanity.com/paper/2410.09456/) (2024)
- [Contextual Biasing of Named-Entities with Large Language Models](https://mlsanity.com/paper/2309.00723/) (2023)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer](https://mlsanity.com/paper/2311.08966/) (2023)
- [Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition](https://mlsanity.com/paper/2301.02736/) (2023)
- [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023)
- [Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR](https://mlsanity.com/paper/2404.10180/) (2024)
- [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024)
- [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023)
- [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023)
- [DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition](https://mlsanity.com/paper/2403.17645/) (2024)
- [Multi-Modal Retrieval For Large Language Model Based Speech Recognition](https://mlsanity.com/paper/2406.09618/) (2024)
- [Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?](https://mlsanity.com/paper/2210.13189/) (2022)
- [Promptformer: Prompted Conformer Transducer for ASR](https://mlsanity.com/paper/2401.07360/) (2024)
- [Locality enhanced dynamic biasing and sampling strategies for contextual ASR](https://mlsanity.com/paper/2401.13146/) (2024)
- [Text Injection for Neural Contextual Biasing](https://mlsanity.com/paper/2406.02921/) (2024)
- [Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach](https://mlsanity.com/paper/2409.09009/) (2024)
- [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024)
- [WCTC-Biasing: Retraining-free Contextual Biasing ASR with Wildcard CTC-based Keyword Spotting and Inter-layer Biasing](https://mlsanity.com/paper/2506.01263/) (2025)

## Abstract (arXiv)

End-to-end (E2E) automatic speech recognition (ASR) systems often have difficulty recognizing uncommon words, that appear infrequently in the training data. One promising method, to improve the recognition accuracy on such rare words, is to latch onto personalized/contextual information at inference. In this work, we present a novel context-aware transformer transducer (CATT) network that improves the state-of-the-art transformer-based ASR system by taking advantage of such contextual signals. Specifically, we propose a multi-head attention-based context-biasing network, which is jointly trained with the rest of the ASR sub-networks. We explore different techniques to encode contextual data and to create the final attention context vectors. We also leverage both BLSTM and pretrained BERT based models to encode contextual data and guide the network training. Using an in-house far-field dataset, we show that CATT, using a BERT based context encoder, improves the word error rate of the baseline transformer transducer and outperforms an existing deep contextual model by 24.2% and 19.4% respectively.