# Improving RNN Transducer Based ASR with Auxiliary Tasks

arXiv: https://arxiv.org/abs/2011.03109 · HTML: https://mlsanity.com/paper/2011.03109/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2020.

## Built on by (most cited first, 27 of 27)

- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [Multi-Head State Space Model for Speech Recognition](https://mlsanity.com/paper/2305.12498/) (2023)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [Non-autoregressive Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/2104.02258/) (2021)
- [Collaborative Training of Acoustic Encoders for Speech Recognition](https://mlsanity.com/paper/2106.08960/) (2021)
- [On lattice-free boosted MMI training of HMM and CTC-based full-context ASR models](https://mlsanity.com/paper/2107.04154/) (2021)
- [Transfer Learning for Robust Low-Resource Children's Speech ASR with Transformers and Source-Filter Warping](https://mlsanity.com/paper/2206.09396/) (2022)
- [Learning a Dual-Mode Speech Recognition Model via Self-Pruning](https://mlsanity.com/paper/2207.11906/) (2022)
- [Sequence Transduction with Graph-based Supervision](https://mlsanity.com/paper/2111.01272/) (2021)
- [Improving CTC-based ASR Models with Gated Interlayer Collaboration](https://mlsanity.com/paper/2205.12462/) (2022)
- [TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models](https://mlsanity.com/paper/2309.01947/) (2023)
- [Memory-Efficient Training of RNN-Transducer with Sampled Softmax](https://mlsanity.com/paper/2203.16868/) (2022)
- [Noisy Training Improves E2E ASR for the Edge](https://mlsanity.com/paper/2107.04677/) (2021)
- [Back from the future: bidirectional CTC decoding using future information in speech recognition](https://mlsanity.com/paper/2110.03326/) (2021)
- [Augmenting text for spoken language understanding with Large Language Models](https://mlsanity.com/paper/2309.09390/) (2023)
- [Pushing the performances of ASR models on English and Spanish accents](https://mlsanity.com/paper/2212.12048/) (2022)
- [Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network](https://mlsanity.com/paper/2309.08489/) (2023)
- [Weak Alignment Supervision from Hybrid Model Improves End-to-end ASR](https://mlsanity.com/paper/2311.14835/) (2023)
- [Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition](https://mlsanity.com/paper/2205.11998/) (2022)
- [Alternating Weak Triphone/BPE Alignment Supervision from Hybrid Model Improves End-to-End ASR](https://mlsanity.com/paper/2402.15594/) (2024)

## Abstract (arXiv)

End-to-end automatic speech recognition (ASR) models with a single neural network have recently demonstrated state-of-the-art results compared to conventional hybrid speech recognizers. Specifically, recurrent neural network transducer (RNN-T) has shown competitive ASR performance on various benchmarks. In this work, we examine ways in which RNN-T can achieve better ASR accuracy via performing auxiliary tasks. We propose (i) using the same auxiliary task as primary RNN-T ASR task, and (ii) performing context-dependent graphemic state prediction as in conventional hybrid modeling. In transcribing social media videos with varying training data size, we first evaluate the streaming ASR performance on three languages: Romanian, Turkish and German. We find that both proposed methods provide consistent improvements. Next, we observe that both auxiliary tasks demonstrate efficacy in learning deep transformer encoders for RNN-T criterion, thus achieving competitive results - 2.0%/4.2% WER on LibriSpeech test-clean/other - as compared to prior top performing models.