# Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition

arXiv: https://arxiv.org/abs/2304.08862 · HTML: https://mlsanity.com/paper/2304.08862/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [A spelling correction model for end-to-end speech recognition](https://mlsanity.com/paper/1902.07178/) (2019)
- [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022)
- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Contextual Speech Recognition with Difficult Negative Training Examples](https://mlsanity.com/paper/1810.12170/) (2018)
- [Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition](https://mlsanity.com/paper/2110.02220/) (2021)
- [Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition](https://mlsanity.com/paper/2010.03192/) (2020)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [No Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models](https://mlsanity.com/paper/1712.01864/) (2017)
- [Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation](https://mlsanity.com/paper/2210.12214/) (2022)

## Built on by (most cited first, 8 of 8)

- [Contextualization of ASR with LLM using phonetic retrieval-based augmentation](https://mlsanity.com/paper/2409.15353/) (2024)
- [Retrieve and Copy: Scaling ASR Personalization to Large Catalogs](https://mlsanity.com/paper/2311.08402/) (2023)
- [Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models](https://mlsanity.com/paper/2312.03632/) (2023)
- [A Multimodal Approach to Device-Directed Speech Detection with Large Language Models](https://mlsanity.com/paper/2403.14438/) (2024)
- [Locality enhanced dynamic biasing and sampling strategies for contextual ASR](https://mlsanity.com/paper/2401.13146/) (2024)
- [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024)
- [Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling](https://mlsanity.com/paper/2509.05908/) (2025)
- [Wiki-En-ASR-Adapt: Large-scale synthetic dataset for English ASR Customization](https://mlsanity.com/paper/2309.17267/) (2023)

## Abstract (arXiv)

This paper presents an extension to train end-to-end Context-Aware Transformer Transducer ( CATT ) models by using a simple, yet efficient method of mining hard negative phrases from the latent space of the context encoder. During training, given a reference query, we mine a number of similar phrases using approximate nearest neighbour search. These sampled phrases are then used as negative examples in the context list alongside random and ground truth contextual information. By including approximate nearest neighbour phrases (ANN-P) in the context list, we encourage the learned representation to disambiguate between similar, but not identical, biasing phrases. This improves biasing accuracy when there are several similar phrases in the biasing inventory. We carry out experiments in a large-scale data regime obtaining up to 7% relative word error rate reductions for the contextual portion of test data. We also extend and evaluate CATT approach in streaming applications.