mlsanityData to 7 Oct 2026

Speech Recognition Rescoring with Large Speech-Text Foundation Models

arXiv 2409.16654 ↗First posted 25 Sep 2024eess.AS cs.CL cs.SDcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 39 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

31 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Spirit LM: Interleaved Spoken and Written Language Model8 Feb
  2. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 29

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  8. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  9. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  10. 2022Generative Spoken Dialogue Language Modelingcited by 125
  11. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  12. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
17 more
  1. 2023Textually Pretrained Speech Language Modelscited by 99
  2. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  3. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  4. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  5. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  6. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
  7. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  8. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  9. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  10. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  11. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16
  12. 2019Audio-attention discriminative language model for ASR rescoringcited by 13
  13. 2019An Empirical Study of Efficient ASR Rescoring with Transformerscited by 11
  14. 2023Scaling Laws for Discriminative Speech Recognition Rescoring Modelscited by 5
  15. 2023Distillation Strategies for Discriminative Speech Recognition Rescoringcited by 2
  16. 2023Personalization for BERT-based Discriminative Speech Recognition Rescoringcited by 2
  17. 2023Discriminative Speech Recognition Rescoring with Pre-trained Language Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognitioncited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing
  2. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition
  3. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems
  4. 2023LLaSM: Large Language and Speech Model
  5. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  6. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputs
  7. 2025Assessment of L2 Oral Proficiency using Speech Large Language Models
  8. 2023A Comprehensive Overview of Large Language Models