mlsanityData to 7 Oct 2026

Improving the fusion of acoustic and text representations in RNN-T

arXiv 2201.10240 ↗First posted 25 Jan 2022eess.AS cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

The recurrent neural network transducer (RNN-T) has recently become the mainstream end-to-end approach for streaming automatic speech recognition (ASR). To estimate the output distributions over subword units, RNN-T uses a fully connected layer as the joint network to fuse the acoustic representations extracted using the acoustic encoder with the text representations obtained using the prediction network based on the previous subword units. In this paper, we propose to use gating, bilinear pooling, and a combination of them in the joint network to produce more expressive representations to feed into the output layer. A regularisation method is also proposed to enable better acoustic encoder training by reducing the gradients back-propagated into the prediction network at the beginning of RNN-T training. Experimental results on a multilingual ASR setting for voice search over nine languages show that the joint use of the proposed methods can result in 4%--5% relative word error rate reductions with only a few million extra parameters.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 25 Jan 2022.

Builds on

32 of 45 references.

References · 32

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  3. 2015Listen, Attend and Spellcited by 542
  4. 2015Attention-Based Models for Speech Recognitioncited by 400
  5. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  6. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  7. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  8. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  9. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  10. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  11. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  12. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
20 more
  1. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  2. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  3. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  4. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  5. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  6. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  7. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  8. 2019SpecAugment on Large Scale Datasetscited by 60
  9. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 51
  10. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  11. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  12. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  13. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  14. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  15. 2020A New Training Pipeline for an Improved Neural Transducercited by 34
  16. 2021Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognitioncited by 32
  17. 2021Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge Devicescited by 18
  18. 2015Highway Long Short-Term Memory RNNs for Distant Speech Recognitioncited by 17
  19. 2020Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Datacited by 14
  20. 2020Combination of Deep Speaker Embeddings for Diarisationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 0; 2025: 1; 2026: 1.

11 papers.

  1. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  2. 2022Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generatorcited by 11
  3. 2023Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generatorcited by 6
  4. 2023CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognitioncited by 6
  5. 2022Adaptive Discounting of Implicit Language Models in RNN-Transducerscited by 5
  6. 2023Improving RNN-Transducers with Acoustic LookAheadcited by 4
  7. 2023UML: A Universal Monolingual Output Layer for Multilingual ASRcited by 3
  8. 2022Contextual-Utterance Training for Automatic Speech Recognitioncited by 1
  9. 2026Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processingcited by 1
  10. 2023Multi-View Frequency-Attention Alternative to CNN Frontends for Automatic Speech Recognitioncited by 0
  11. 2025SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translationcited by 0