mlsanityData to 7 Oct 2026

Memory-Efficient Training of RNN-Transducer with Sampled Softmax

arXiv 2203.16868 ↗First posted 31 Mar 2022eess.AS cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

RNN-Transducer has been one of promising architectures for end-to-end automatic speech recognition. Although RNN-Transducer has many advantages including its strong accuracy and streaming-friendly property, its high memory consumption during training has been a critical problem for development. In this work, we propose to apply sampled softmax to RNN-Transducer, which requires only a small subset of vocabulary during training thus saves its memory consumption. We further extend sampled softmax to optimize memory consumption for a minibatch, and employ distributions of auxiliary CTC losses for sampling vocabulary to improve model accuracy. We present experimental results on LibriSpeech, AISHELL-1, and CSJ-APS, where sampled softmax greatly reduces memory consumption and still maintains the accuracy of the baseline model.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

38 of 43 references.

References · 38

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  5. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  7. 2015Listen, Attend and Spellcited by 542
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  9. 2015Attention-Based Models for Speech Recognitioncited by 400
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  11. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  12. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
26 more
  1. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  2. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  3. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
  4. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  5. 2017Monotonic Chunkwise Attentioncited by 82
  6. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  7. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  8. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  9. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  10. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  11. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  12. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  13. 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
  14. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  15. 2020Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognitioncited by 34
  16. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 29
  17. 2020Improving RNN Transducer Based ASR with Auxiliary Taskscited by 27
  18. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  19. 2019Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networkscited by 24
  20. 2020Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Convertercited by 16
  21. 2022A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategiescited by 15
  22. 2020Benchmarking LF-MMI, CTC and RNN-T Criteria for Streaming ASRcited by 11
  23. 2020Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognitioncited by 9
  24. 2018Exploring RNN-Transducer for Chinese Speech Recognitioncited by 8
  25. 2020Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognitioncited by 4
  26. 2021A comparison of streaming models and data augmentation methods for robust speech recognitioncited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1.

3 papers.

  1. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
  2. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  3. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8