# Pruned RNN-T for fast, memory-efficient ASR training

arXiv: https://arxiv.org/abs/2206.13236 · HTML: https://mlsanity.com/paper/2206.13236/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [Benchmarking LF-MMI, CTC and RNN-T Criteria for Streaming ASR](https://mlsanity.com/paper/2011.04785/) (2020)

## Built on by (most cited first, 30 of 46)

- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement](https://mlsanity.com/paper/2406.11546/) (2024)
- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Accelerating RNN-T Training and Inference Using CTC guidance](https://mlsanity.com/paper/2210.16481/) (2022)
- [CR-CTC: Consistency regularization on CTC for improved speech recognition](https://mlsanity.com/paper/2410.05101/) (2024)
- [Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks](https://mlsanity.com/paper/2210.07499/) (2022)
- [Fast and parallel decoding for transducer](https://mlsanity.com/paper/2211.00484/) (2022)
- [Multi-blank Transducers for Speech Recognition](https://mlsanity.com/paper/2211.03541/) (2022)
- [Updated Corpora and Benchmarks for Long-Form Speech Recognition](https://mlsanity.com/paper/2309.15013/) (2023)
- [Blank-regularized CTC for Frame Skipping in Neural Transducer](https://mlsanity.com/paper/2305.11558/) (2023)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [Powerful and Extensible WFST Framework for RNN-Transducer Losses](https://mlsanity.com/paper/2303.10384/) (2023)
- [Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction](https://mlsanity.com/paper/2311.02898/) (2023)
- [Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation](https://mlsanity.com/paper/2211.00508/) (2022)
- [CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition](https://mlsanity.com/paper/2307.14132/) (2023)
- [Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation](https://mlsanity.com/paper/2210.12214/) (2022)
- [SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation](https://mlsanity.com/paper/2403.08196/) (2024)
- [Joint Beam Search Integrating CTC, Attention, and Transducer Decoders](https://mlsanity.com/paper/2406.02950/) (2024)
- [VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining](https://mlsanity.com/paper/2505.21527/) (2025)
- [TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR](https://mlsanity.com/paper/2407.04444/) (2024)
- [Clean Label Attacks against SLU Systems](https://mlsanity.com/paper/2409.08985/) (2024)
- [Early Attentive Sparsification Accelerates Neural Speech Transcription](https://mlsanity.com/paper/2506.15912/) (2025)
- [Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation](https://mlsanity.com/paper/2211.16270/) (2022)
- [Bayes Risk Transducer: Transducer with Controllable Alignment Prediction](https://mlsanity.com/paper/2308.10107/) (2023)
- [Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction](https://mlsanity.com/paper/2401.01498/) (2024)
- [Self-consistent context aware conformer transducer for speech recognition](https://mlsanity.com/paper/2402.06592/) (2024)
- [The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge](https://mlsanity.com/paper/2404.06079/) (2024)
- [Romanization Encoding For Multilingual ASR](https://mlsanity.com/paper/2407.04368/) (2024)

## Abstract (arXiv)

The RNN-Transducer (RNN-T) framework for speech recognition has been growing in popularity, particularly for deployed real-time ASR systems, because it combines high accuracy with naturally streaming recognition. One of the drawbacks of RNN-T is that its loss function is relatively slow to compute, and can use a lot of memory. Excessive GPU memory usage can make it impractical to use RNN-T loss in cases where the vocabulary size is large: for example, for Chinese character-based ASR. We introduce a method for faster and more memory-efficient RNN-T loss computation. We first obtain pruning bounds for the RNN-T recursion using a simple joiner network that is linear in the encoder and decoder embeddings; we can evaluate this without using much memory. We then use those pruning bounds to evaluate the full, non-linear joiner network.