# Variable Attention Masking for Configurable Transformer Transducer Speech Recognition arXiv: https://arxiv.org/abs/2211.01438 ยท HTML: https://mlsanity.com/paper/2211.01438/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 32 references that are in the map. - First posted on arXiv 2 Nov 2022. ## Builds on (most cited first, 24 of 24) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019) - [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020) - [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021) - [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021) - [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020) - [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020) - [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020) - [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020) - [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020) - [CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings](https://mlsanity.com/paper/2106.03143/) (2021) - [Transformer ASR with Contextual Block Processing](https://mlsanity.com/paper/1910.07204/) (2019) - [Synchronous Transformers for End-to-End Speech Recognition](https://mlsanity.com/paper/1912.02958/) (2019) - [Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition](https://mlsanity.com/paper/2010.03192/) (2020) - [Relative Positional Encoding for Speech Recognition and Direct Translation](https://mlsanity.com/paper/2005.09940/) (2020) - [Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition](https://mlsanity.com/paper/2008.05750/) (2020) - [Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data](https://mlsanity.com/paper/2010.12096/) (2020) - [Reducing Streaming ASR Model Delay with Self Alignment](https://mlsanity.com/paper/2105.05005/) (2021) - [Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2107.01269/) (2021) - [Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition](https://mlsanity.com/paper/2006.01712/) (2020) - [Streaming parallel transducer beam search with fast-slow cascaded encoders](https://mlsanity.com/paper/2203.15773/) (2022) ## Built on by (most cited first, 14 of 14) - [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024) - [Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition](https://mlsanity.com/paper/2304.08862/) (2023) - [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023) - [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024) - [Cross-lingual Knowledge Transfer and Iterative Pseudo-labeling for Low-Resource Speech Recognition with Transducers](https://mlsanity.com/paper/2305.13652/) (2023) - [Bigger is not Always Better: The Effect of Context Size on Speech Pre-Training](https://mlsanity.com/paper/2312.01515/) (2023) - [Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization](https://mlsanity.com/paper/2505.24229/) (2025) - [Adaptive Turn-Taking for Real-time Multi-Party Voice Agents](https://mlsanity.com/paper/2606.13544/) (2026) - [Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper](https://mlsanity.com/paper/2409.13499/) (2024) - [Unifying Streaming and Non-streaming Zipformer-based ASR](https://mlsanity.com/paper/2506.14434/) (2025) - [Neural Language Model Pruning for Automatic Speech Recognition](https://mlsanity.com/paper/2310.03424/) (2023) - [Segmental Attention Decoding With Long Form Acoustic Encodings](https://mlsanity.com/paper/2512.14652/) (2025) - [Activation-Based Active Learning for In-Context Learning: Challenges and Insights](https://mlsanity.com/paper/2606.05134/) (2026) - [Reading Less While Writing: A Closed-Form Bandwidth Dial for Streaming Multimodal Decoders](https://mlsanity.com/paper/2609.20845/) (2026) ## Abstract (arXiv) This work studies the use of attention masking in transformer transducer based speech recognition for building a single configurable model for different deployment scenarios. We present a comprehensive set of experiments comparing fixed masking, where the same attention mask is applied at every frame, with chunked masking, where the attention mask for each frame is determined by chunk boundaries, in terms of recognition accuracy and latency. We then explore the use of variable masking, where the attention masks are sampled from a target distribution at training time, to build models that can work in different configurations. Finally, we investigate how a single configurable model can be used to perform both first pass streaming recognition and second pass acoustic rescoring. Experiments show that chunked masking achieves a better accuracy vs latency trade-off compared to fixed masking, both with and without FastEmit. We also show that variable masking improves the accuracy by up to 8% relative in the acoustic re-scoring scenario.