mlsanityData to 7 Oct 2026

Accelerating RNN-T Training and Inference Using CTC guidance

arXiv 2210.16481 ↗First posted 29 Oct 2022eess.AS cs.CL cs.SDcited by 17

Not on the map. Why?

Abstract (arXiv)

We propose a novel method to accelerate training and inference process of recurrent neural network transducer (RNN-T) based on the guidance from a co-trained connectionist temporal classification (CTC) model. We made a key assumption that if an encoder embedding frame is classified as a blank frame by the CTC model, it is likely that this frame will be aligned to blank for all the partial alignments or hypotheses in RNN-T and it can be discarded from the decoder input. We also show that this frame reduction operation can be applied in the middle of the encoder, which result in significant speed up for the training and inference in RNN-T. We further show that the CTC alignment, a by-product of the CTC decoder, can also be used to perform lattice reduction for RNN-T during training. Our method is evaluated on the Librispeech and SpeechStew tasks. We demonstrate that the proposed method is able to accelerate the RNN-T inference by 2.2 times with similar or slightly better word error rates (WER).

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

22 of 31 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  6. 2015Listen, Attend and Spellcited by 542
  7. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  8. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  9. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  10. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  11. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  12. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
10 more
  1. 2017Monotonic Chunkwise Attentioncited by 82
  2. 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
  3. 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
  4. 2016Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognitioncited by 33
  5. 2015Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognitioncited by 26
  6. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  7. 2021Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge Devicescited by 18
  8. 2021FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularizationcited by 11
  9. 2022LegoNN: Building Modular Encoder-Decoder Modelscited by 9
  10. 2020Modular End-to-end Automatic Speech Recognition Framework for Acoustic-to-word Modelcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 7; 2025: 4.

17 papers.

Sort:

By area:

  1. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  2. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  3. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9
  4. 2023Blank-regularized CTC for Frame Skipping in Neural Transducercited by 8
  5. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  6. 2023CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognitioncited by 6
  7. 2024Alignment-Free Training for Transducer-based Multi-Talker ASRcited by 4
  8. 2025Pushing the Limits of Beam Search Decoding for Transducer-based ASR modelscited by 4
  9. 2022Blank Collapse: Compressing CTC emission for the faster decodingcited by 3
  10. 2023Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognitioncited by 2
  11. 2024Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognitioncited by 2
  12. 2024Lightweight Transducer Based on Frame-Level Criterioncited by 1
5 more
  1. 2025RNN-Transducer-based Losses for Speech Recognition on Noisy Targetscited by 1
  2. 2024Coupling Speech Encoders with Downstream Text Modelscited by 0
  3. 2024Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholdingcited by 0
  4. 2025FLToP CTC: Frame-Level Token Pruning via Relative Threshold for Efficient and Memory-Saving Decoding on Diverse Platformscited by 0
  5. 2025End-to-end Speech Recognition with similar length speech and textcited by 0