# Modular Hybrid Autoregressive Transducer

arXiv: https://arxiv.org/abs/2210.17049 · HTML: https://mlsanity.com/paper/2210.17049/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 61 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 30 of 32)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020)
- [Factorized Neural Transducer for Efficient Language Model Adaptation](https://mlsanity.com/paper/2110.01500/) (2021)
- [Librispeech Transducer Model with Internal Language Model Prior Correction](https://mlsanity.com/paper/2104.03006/) (2021)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Speaker-Invariant Training via Adversarial Learning](https://mlsanity.com/paper/1804.00732/) (2018)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network](https://mlsanity.com/paper/2104.11127/) (2021)
- [Improving Performance of End-to-End ASR on Numeric Sequences](https://mlsanity.com/paper/1907.01372/) (2019)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [Invariant Representations for Noisy Speech Recognition](https://mlsanity.com/paper/1612.01928/) (2016)
- [Speaker Adaptation for Attention-Based End-to-End Speech Recognition](https://mlsanity.com/paper/1911.03762/) (2019)
- [Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition](https://mlsanity.com/paper/1711.08010/) (2017)

## Built on by (most cited first, 17 of 17)

- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator](https://mlsanity.com/paper/2302.14036/) (2023)
- [JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition](https://mlsanity.com/paper/2302.08583/) (2023)
- [NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding](https://mlsanity.com/paper/2505.22857/) (2025)
- [Understanding Shared Speech-Text Representations](https://mlsanity.com/paper/2304.14514/) (2023)
- [Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning](https://mlsanity.com/paper/2412.06967/) (2024)
- [Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition](https://mlsanity.com/paper/2412.16464/) (2024)
- [Pushing the Limits of Beam Search Decoding for Transducer-based ASR models](https://mlsanity.com/paper/2506.00185/) (2025)
- [Decoupled Structure for Improved Adaptability of End-to-End Models](https://mlsanity.com/paper/2308.13345/) (2023)
- [Effective internal language model training and fusion for factorized transducer model](https://mlsanity.com/paper/2404.01716/) (2024)
- [Text Injection for Neural Contextual Biasing](https://mlsanity.com/paper/2406.02921/) (2024)
- [External Language Model Integration for Factorized Neural Transducers](https://mlsanity.com/paper/2305.17304/) (2023)
- [Improved Factorized Neural Transducer Model For text-only Domain Adaptation](https://mlsanity.com/paper/2309.09524/) (2023)
- [Text Injection for Capitalization and Turn-Taking Prediction in Speech Models](https://mlsanity.com/paper/2308.07395/) (2023)
- [Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation](https://mlsanity.com/paper/2309.07369/) (2023)
- [Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers](https://mlsanity.com/paper/2502.05232/) (2025)
- [RNN-Transducer-based Losses for Speech Recognition on Noisy Targets](https://mlsanity.com/paper/2504.06963/) (2025)

## Abstract (arXiv)

Text-only adaptation of a transducer model remains challenging for end-to-end speech recognition since the transducer has no clearly separated acoustic model (AM), language model (LM) or blank model. In this work, we propose a modular hybrid autoregressive transducer (MHAT) that has structurally separated label and blank decoders to predict label and blank distributions, respectively, along with a shared acoustic encoder. The encoder and label decoder outputs are directly projected to AM and internal LM scores and then added to compute label posteriors. We train MHAT with an internal LM loss and a HAT loss to ensure that its internal LM becomes a standalone neural LM that can be effectively adapted to text. Moreover, text adaptation of MHAT fosters a much better LM fusion than internal LM subtraction-based methods. On Google's large-scale production data, a multi-domain MHAT adapted with 100B sentences achieves relative WER reductions of up to 12.4% without LM fusion and 21.5% with LM fusion from 400K-hour trained HAT.