# Factorized Neural Transducer for Efficient Language Model Adaptation

arXiv: https://arxiv.org/abs/2110.01500 · HTML: https://mlsanity.com/paper/2110.01500/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017)
- [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019)
- [Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems](https://mlsanity.com/paper/2011.11564/) (2020)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network](https://mlsanity.com/paper/2104.11127/) (2021)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [Advancing Acoustic-to-Word CTC Model](https://mlsanity.com/paper/1803.05566/) (2018)

## Built on by (most cited first, 30 of 32)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition](https://mlsanity.com/paper/2306.16007/) (2023)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [Modular Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2210.17049/) (2022)
- [Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models](https://mlsanity.com/paper/2212.01992/) (2022)
- [JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition](https://mlsanity.com/paper/2302.08583/) (2023)
- [ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correction](https://mlsanity.com/paper/2310.05129/) (2023)
- [LongFNT: Long-form Speech Recognition with Factorized Neural Transducer](https://mlsanity.com/paper/2211.09412/) (2022)
- [Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation](https://mlsanity.com/paper/2305.17846/) (2023)
- [Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation](https://mlsanity.com/paper/2406.04541/) (2024)
- [Improving RNN-Transducers with Acoustic LookAhead](https://mlsanity.com/paper/2307.05006/) (2023)
- [Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning](https://mlsanity.com/paper/2412.06967/) (2024)
- [Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition](https://mlsanity.com/paper/2412.16464/) (2024)
- [Decoupled Structure for Improved Adaptability of End-to-End Models](https://mlsanity.com/paper/2308.13345/) (2023)
- [Effective internal language model training and fusion for factorized transducer model](https://mlsanity.com/paper/2404.01716/) (2024)
- [Text Injection for Neural Contextual Biasing](https://mlsanity.com/paper/2406.02921/) (2024)
- [NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training](https://mlsanity.com/paper/2409.08680/) (2024)
- [External Language Model Integration for Factorized Neural Transducers](https://mlsanity.com/paper/2305.17304/) (2023)
- [Improved Factorized Neural Transducer Model For text-only Domain Adaptation](https://mlsanity.com/paper/2309.09524/) (2023)
- [Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognition](https://mlsanity.com/paper/2403.08258/) (2024)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)
- [Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation](https://mlsanity.com/paper/2309.07369/) (2023)
- [Lightweight Transducer Based on Frame-Level Criterion](https://mlsanity.com/paper/2409.13698/) (2024)
- [Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholding](https://mlsanity.com/paper/2409.20313/) (2024)
- [Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers](https://mlsanity.com/paper/2502.05232/) (2025)
- [PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding](https://mlsanity.com/paper/2506.11064/) (2025)
- [Bilingual Streaming ASR with Grapheme units and Auxiliary Monolingual Loss](https://mlsanity.com/paper/2308.06327/) (2023)
- [Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer](https://mlsanity.com/paper/2309.07648/) (2023)
- [PARCO: Phoneme-Augmented Robust Contextual ASR via Contrastive Entity Disambiguation](https://mlsanity.com/paper/2509.04357/) (2025)

## Abstract (arXiv)

In recent years, end-to-end (E2E) based automatic speech recognition (ASR) systems have achieved great success due to their simplicity and promising performance. Neural Transducer based models are increasingly popular in streaming E2E based ASR systems and have been reported to outperform the traditional hybrid system in some scenarios. However, the joint optimization of acoustic model, lexicon and language model in neural Transducer also brings about challenges to utilize pure text for language model adaptation. This drawback might prevent their potential applications in practice. In order to address this issue, in this paper, we propose a novel model, factorized neural Transducer, by factorizing the blank and vocabulary prediction, and adopting a standalone language model for the vocabulary prediction. It is expected that this factorization can transfer the improvement of the standalone language model to the Transducer for speech recognition, which allows various language model adaptation techniques to be applied. We demonstrate that the proposed factorized neural Transducer yields 15% to 20% WER improvements when out-of-domain text data is used for language model adaptation, at the cost of a minor degradation in WER on a general test set.