mlsanityData to 7 Oct 2026

Tied & Reduced RNN-T Decoder

arXiv 2109.07513 ↗First posted 15 Sep 2021cs.CL cs.LG cs.SDcited by 35

Not on the map. Why?

Abstract (arXiv)

Previous works on the Recurrent Neural Network-Transducer (RNN-T) models have shown that, under some conditions, it is possible to simplify its prediction network with little or no loss in recognition accuracy (arXiv:2003.07705 [eess.AS], [2], arXiv:2012.06749 [cs.CL]). This is done by limiting the context size of previous labels and/or using a simpler architecture for its layers instead of LSTMs. The benefits of such changes include reduction in model size, faster inference and power savings, which are all useful for on-device applications. In this work, we study ways to make the RNN-T decoder (prediction network + joint network) smaller and faster without degradation in recognition performance. Our prediction network performs a simple weighted averaging of the input embeddings, and shares its embedding matrix weights with the joint network's output layer (a.k.a. weight tying, commonly used in language modeling arXiv:1611.01462 [cs.LG]). This simple design, when used in conjunction with additional Edit-based Minimum Bayes Risk (EMBR) training, reduces the RNN-T Decoder from 23M parameters to just 2M, without affecting word-error rate (WER).

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

24 of 35 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  5. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  6. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  7. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  8. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  9. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  10. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  11. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  12. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
12 more
  1. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  2. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  3. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  4. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  5. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  6. 2019SpecAugment on Large Scale Datasetscited by 60
  7. 2020Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognitioncited by 34
  8. 2016Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognitioncited by 33
  9. 2020Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Mergingcited by 26
  10. 2019Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognitioncited by 23
  11. 2021Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge Devicescited by 18
  12. 2021Echo State Speech Recognitioncited by 11

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 16; 2023: 12; 2024: 5.

35 papers.

Sort:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2023End-to-End Speech Recognition: A Surveycited by 127
  3. 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
  4. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  5. 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 22
  6. 2022E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASRcited by 17
  7. 2022Modular Hybrid Autoregressive Transducercited by 17
  8. 2022Turn-Taking Prediction for Natural Conversational Speechcited by 15
  9. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  10. 2021Deliberation of Streaming RNN-Transducer by Non-autoregressive Decodingcited by 13
  11. 2022Improving Rare Word Recognition with LM-aware MWER Trainingcited by 12
  12. 2022A Unified Cascaded Encoder ASR Model for Dynamic Model Sizescited by 11
23 more
  1. 2022Improving Deliberation by Text-Only and Semi-Supervised Trainingcited by 10
  2. 2022Detecting Unintended Memorization in Language-Model-Fused ASRcited by 9
  3. 2022A Language Agnostic Multilingual Streaming On-Device ASR Systemcited by 9
  4. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  5. 2023Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithmcited by 9
  6. 2023Modular Domain Adaptation for Conformer-Based Streaming ASRcited by 7
  7. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognitioncited by 6
  8. 2022LongFNT: Long-form Speech Recognition with Factorized Neural Transducercited by 6
  9. 2023CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognitioncited by 6
  10. 2024Breaking Down Power Barriers in On-Device Streaming ASR: Insights and Solutionscited by 5
  11. 2024Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Modelscited by 5
  12. 2024Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASRcited by 5
  13. 2022VQ-T: RNN Transducers using Vector-Quantized Prediction Network Statescited by 3
  14. 2022Streaming Intended Query Detection using E2E Modeling for Continued Conversationcited by 3
  15. 2023Lego-Features: Exporting modular encoder features for streaming and deliberation ASRcited by 3
  16. 2024Text Injection for Neural Contextual Biasingcited by 3
  17. 2023Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakerscited by 2
  18. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  19. 2022On the Prediction Network Architecture in RNN-T for ASRcited by 1
  20. 2023Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Networkcited by 1
  21. 2023Dual Learning for Large Vocabulary On-Device ASRcited by 0
  22. 2023Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASRcited by 0
  23. 2023Partial Rewriting for Multi-Stage ASRcited by 0