mlsanityData to 7 Oct 2026

USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder

arXiv 2202.06045 ↗First posted 12 Feb 2022cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Improving end-to-end speech recognition by incorporating external text data has been a longstanding research topic. There has been a recent focus on training E2E ASR models that get the performance benefits of external text data without incurring the extra cost of evaluating an external language model at inference time. In this work, we propose training ASR model jointly with a set of text-to-text auxiliary tasks with which it shares a decoder and parts of the encoder. When we jointly train ASR and masked language model with the 960-hour Librispeech and Opensubtitles data respectively, we observe WER reductions of 16% and 20% on test-other and test-clean respectively over an ASR-only baseline without any extra cost at inference time, and reductions of 6% and 8% compared to a stronger MUTE-L baseline which trains the decoder with the same text data as our model. We achieve further improvements when we train masked language model on Librispeech data or when we use machine translation as the auxiliary task, without significantly sacrificing performance on the task itself.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 12 Feb 2022.

Builds on

19 of 24 references.

References · 19

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2015Listen, Attend and Spellcited by 542
  5. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  6. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  7. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  8. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  9. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  10. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  11. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  12. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
7 more
  1. 2019Generating Synthetic Audio Data for Attention-Based Speech Recognition Systemscited by 46
  2. 2020Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Trainingcited by 38
  3. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  4. 2021SynthASR: Unlocking Synthetic Data for Speech Recognitioncited by 34
  5. 2020Multitask Training with Text Data for End-to-End Speech Recognitioncited by 19
  6. 2020A Convolutional Deep Markov Model for Unsupervised Speech Representation Learningcited by 18
  7. 2021EAT: Enhanced ASR-TTS for Self-supervised Speech Recognitioncited by 13

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 1.

7 papers.

  1. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  2. 2023On-the-fly Text Retrieval for End-to-End ASR Adaptationcited by 4
  3. 2024Speculative Speech Recognition by Audio-Prefixed Low-Rank Adaptation of Language Modelscited by 3
  4. 2023A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scalecited by 1
  5. 2025Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Trackingcited by 1
  6. 2023End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representationscited by 0
  7. 2024Written Term Detection Improves Spoken Term Detectioncited by 0