mlsanityData to 14 Oct 2026

Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning

arXiv 2412.06967 ↗First posted 9 Dec 2024cs.CL cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

The advent of Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR). Prompting LLM with audio embeddings to generate transcriptions becomes the new state-of-the-art ASR. Despite LLMs being trained with an extensive amount of text corpora, high-quality domain-specific text data can still significantly enhance ASR performance on domain adaptation tasks. Although LLM-based ASR can naturally incorporate more text corpora by fine-tuning the LLM decoder, fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge. To mitigate this issue, we propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation. Experimental results show that text adaptation with our proposed method achieved a relative up to 9% Word Error Rate (WER) reduction and up to 18% Entity Error Rate (EER) reduction on the target domain compared to the baseline ASR. Combining this with domain-specific Language Model (LM) fusion can further improve the EER by a relative 2-5%

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

26 of 32 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  8. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  9. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  10. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  11. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  12. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
14 more
  1. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  2. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  3. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  4. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 41
  5. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 32
  6. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  7. 2021Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognitioncited by 24
  8. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  9. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  10. 2022Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systemscited by 20
  11. 2021Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Networkcited by 19
  12. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  13. 2022Modular Hybrid Autoregressive Transducercited by 17
  14. 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  2. 2026MEUSLI: a Multilingual Projector for LLM-based ASR and Beyondcited by 1
  3. 2026Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASRcited by 0
  4. 2026CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agentscited by 0