mlsanityData to 14 Oct 2026

Factorized Neural Transducer for Efficient Language Model Adaptation

arXiv 2110.01500 ↗First posted 27 Sep 2021cs.CL eess.AScited by 32

Not on the map. Why?

Abstract (arXiv)

In recent years, end-to-end (E2E) based automatic speech recognition (ASR) systems have achieved great success due to their simplicity and promising performance. Neural Transducer based models are increasingly popular in streaming E2E based ASR systems and have been reported to outperform the traditional hybrid system in some scenarios. However, the joint optimization of acoustic model, lexicon and language model in neural Transducer also brings about challenges to utilize pure text for language model adaptation. This drawback might prevent their potential applications in practice. In order to address this issue, in this paper, we propose a novel model, factorized neural Transducer, by factorizing the blank and vocabulary prediction, and adopting a standalone language model for the vocabulary prediction. It is expected that this factorization can transfer the improvement of the standalone language model to the Transducer for speech recognition, which allows various language model adaptation techniques to be applied. We demonstrate that the proposed factorized neural Transducer yields 15% to 20% WER improvements when out-of-domain text data is used for language model adaptation, at the cost of a minor degradation in WER on a general test set.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

26 of 36 references.

References · 26

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2015Listen, Attend and Spellcited by 543
  3. 2015Attention-Based Models for Speech Recognitioncited by 401
  4. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  5. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  6. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  7. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  8. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 111
  9. 2020Hybrid Autoregressive Transducer (hat)cited by 108
  10. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
  11. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  12. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 96
14 more
  1. 2017Monotonic Chunkwise Attentioncited by 83
  2. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  3. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  4. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 69
  5. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  6. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  7. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 44
  8. 2019Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entitiescited by 41
  9. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  10. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  11. 2019RNN-T For Latency Controlled ASR With Improved Beam Searchcited by 20
  12. 2021Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Networkcited by 19
  13. 2021Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognitioncited by 19
  14. 2018Advancing Acoustic-to-Word CTC Modelcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 12; 2024: 10; 2025: 3; 2026: 2.

32 papers.

Sort:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  2. 2023End-to-End Speech Recognition: A Surveycited by 129
  3. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  4. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  5. 2022Modular Hybrid Autoregressive Transducercited by 17
  6. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
  7. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  8. 2023ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correctioncited by 7
  9. 2022LongFNT: Long-form Speech Recognition with Factorized Neural Transducercited by 6
  10. 2023Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimationcited by 5
  11. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
  12. 2023Improving RNN-Transducers with Acoustic LookAheadcited by 4
20 more
  1. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  2. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  3. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  4. 2024Effective internal language model training and fusion for factorized transducer modelcited by 3
  5. 2024Text Injection for Neural Contextual Biasingcited by 3
  6. 2024NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Trainingcited by 3
  7. 2023External Language Model Integration for Factorized Neural Transducerscited by 2
  8. 2023Improved Factorized Neural Transducer Model For text-only Domain Adaptationcited by 2
  9. 2024Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognitioncited by 2
  10. 2024Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASRcited by 2
  11. 2023Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptationcited by 1
  12. 2024Lightweight Transducer Based on Frame-Level Criterioncited by 1
  13. 2024Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholdingcited by 1
  14. 2025Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducerscited by 1
  15. 2025PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decodingcited by 1
  16. 2023Bilingual Streaming ASR with Grapheme units and Auxiliary Monolingual Losscited by 0
  17. 2023Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducercited by 0
  18. 2025PARCO: Phoneme-Augmented Robust Contextual ASR via Contrastive Entity Disambiguationcited by 0
  19. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  20. 2026Factorized Delayed Streams Modeling for LLM-based Streaming ASRcited by 0