mlsanityData to 7 Oct 2026

Multi-task Language Modeling for Improving Speech Recognition of Rare Words

arXiv 2011.11715 ↗First posted 23 Nov 2020cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

End-to-end automatic speech recognition (ASR) systems are increasingly popular due to their relative architectural simplicity and competitive performance. However, even though the average accuracy of these systems may be high, the performance on rare content words often lags behind hybrid ASR systems. To address this problem, second-pass rescoring is often applied leveraging upon language modeling. In this paper, we propose a second-pass system with multi-task learning, utilizing semantic targets (such as intent and slot prediction) to improve speech recognition performance. We show that our rescoring model trained with these additional tasks outperforms the baseline rescoring model, trained with only the language modeling task, by 1.4% on a general test and by 2.6% on a rare word test set in terms of word-error-rate relative (WERR). Our best ASR system with multi-task LM shows 4.6% WERR deduction compared with RNN Transducer only ASR baseline for rare words recognition.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 5; 2023: 4; 2024: 8; 2025: 1.

21 papers.

Sort:

By area:

  1. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  2. 2022When BERT Meets Quantum Temporal Convolution Learning for Text Classification in Heterogeneous Computingcited by 40
  3. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  4. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  5. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  6. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  7. 2021Training a Resilient Q-Network against Observational Interferencecited by 9
  8. 2021ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modelingcited by 7
  9. 2024Entropy-Reinforced Planning with Large Language Models for Drug Discoverycited by 6
  10. 2021A Lottery Ticket Hypothesis Framework for Low-Complexity Device-Robust Neural Acoustic Scene Classificationcited by 4
  11. 2023Parameter-Efficient Learning for Text-to-Speech Accent Adaptationcited by 4
  12. 2024Evolutionary Prompt Design for LLM-Based Post-ASR Error Correctioncited by 3
9 more
  1. 2024Full-text Error Correction for Chinese Speech Recognition with Large Language Modelcited by 3
  2. 2024Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approachcited by 3
  3. 2022Mitigating Closed-model Adversarial Examples with Bayesian Neural Modeling for Enhanced End-to-End Speech Recognitioncited by 2
  4. 2022Exploring Sequence-to-Sequence Transformer-Transducer Models for Keyword Spottingcited by 2
  5. 2022An Experimental Study on Private Aggregation of Teacher Ensemble Learning for End-to-End Speech Recognitioncited by 1
  6. 2022Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognitioncited by 1
  7. 2023Improving Rare Words Recognition through Homophone Extension and Unified Writing for Low-resource Cantonese Speech Recognitioncited by 1
  8. 2024Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognitioncited by 1
  9. 2025Chain of Correction for Full-text Speech Recognition with Large Language Modelscited by 0