# Multitask Training with Text Data for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2010.14318 · HTML: https://mlsanity.com/paper/2010.14318/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers](https://mlsanity.com/paper/2211.02809/) (2022)
- [USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder](https://mlsanity.com/paper/2202.06045/) (2022)
- [Joint Audio/Text Training for Transformer Rescorer of Streaming Speech Recognition](https://mlsanity.com/paper/2211.00174/) (2022)
- [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023)
- [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021)
- [Speculative Speech Recognition by Audio-Prefixed Low-Rank Adaptation of Language Models](https://mlsanity.com/paper/2407.04641/) (2024)
- [Augmenting text for spoken language understanding with Large Language Models](https://mlsanity.com/paper/2309.09390/) (2023)
- [A Deliberation-based Joint Acoustic and Text Decoder](https://mlsanity.com/paper/2303.15293/) (2023)
- [Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processing](https://mlsanity.com/paper/2602.22522/) (2026)
- [End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations](https://mlsanity.com/paper/2308.08027/) (2023)
- [SememeASR: Boosting Performance of End-to-End Speech Recognition against Domain and Long-Tailed Data Shift with Sememe Semantic Knowledge](https://mlsanity.com/paper/2309.01437/) (2023)
- [Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representation](https://mlsanity.com/paper/2309.02459/) (2023)
- [Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translation](https://mlsanity.com/paper/2406.10276/) (2024)
- [Written Term Detection Improves Spoken Term Detection](https://mlsanity.com/paper/2407.04601/) (2024)
- [LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration](https://mlsanity.com/paper/2601.15397/) (2026)
- [Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving](https://mlsanity.com/paper/2607.01733/) (2026)

## Abstract (arXiv)

We propose a multitask training method for attention-based end-to-end speech recognition models. We regularize the decoder in a listen, attend, and spell model by multitask training it on both audio-text and text-only data. Trained on the 100-hour subset of LibriSpeech, the proposed method, without requiring an additional language model, leads to an 11% relative performance improvement over the baseline and approaches the performance of language model shallow fusion on the test-clean evaluation set. We observe a similar trend on the whole 960-hour LibriSpeech training set. Analyses of different types of errors and sample output sentences demonstrate that the proposed method can incorporate language level information, suggesting its effectiveness in real-world applications.