mlsanityData to 7 Oct 2026

Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models

arXiv 2212.01992 ↗First posted 5 Dec 2022cs.CL cs.SD eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability. However, it is challenging to adapt it with text-only data. Factorized neural transducer (FNT) model was proposed to mitigate this problem. The improved adaptation ability of FNT on text-only adaptation data came at the cost of lowered accuracy compared to the standard neural transducer model. We propose several methods to improve the performance of the FNT model. They are: adding CTC criterion during training, adding KL divergence loss during adaptation, using a pre-trained language model to seed the vocabulary predictor, and an efficient adaptation approach by interpolating the vocabulary predictor with the n-gram language model. A combination of these approaches results in a relative word-error-rate reduction of 9.48% from the standard FNT model. Furthermore, n-gram interpolation with the vocabulary predictor improves the adaptation speed hugely with satisfactory adaptation performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

27 of 37 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  4. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  5. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  6. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  7. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  8. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  9. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  10. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  11. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  12. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
15 more
  1. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  2. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  3. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  4. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  5. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  6. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  7. 2019Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entitiescited by 41
  8. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  9. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  10. 2021SynthASR: Unlocking Synthetic Data for Speech Recognitioncited by 34
  11. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 31
  12. 2021On Addressing Practical Challenges for RNN-Transducercited by 19
  13. 2021Language model fusion for streaming end to end speech recognitioncited by 7
  14. 2020Improving accuracy of rare words for RNN-Transducer through unigram shallow fusioncited by 6
  15. 2022Adaptive Discounting of Implicit Language Models in RNN-Transducerscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 5; 2025: 1.

11 papers.

  1. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  2. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  3. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  4. 2024Effective internal language model training and fusion for factorized transducer modelcited by 3
  5. 2024NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Trainingcited by 3
  6. 2023External Language Model Integration for Factorized Neural Transducerscited by 2
  7. 2023Improved Factorized Neural Transducer Model For text-only Domain Adaptationcited by 2
  8. 2023Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptationcited by 1
  9. 2024Lightweight Transducer Based on Frame-Level Criterioncited by 1
  10. 2025Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Datacited by 1
  11. 2024Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholdingcited by 0