mlsanityData to 14 Oct 2026

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

arXiv 2604.06487 ↗First posted 7 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encoder to a large language model via a projection module, enabling adaptation with text-only data. However, this introduces a modality gap, as the LLM is not exposed to the noisy representations produced by the speech projector. We investigate whether small amounts of speech can mitigate this mismatch. We compare three strategies: text-only adaptation, paired speech-text adaptation, and mixed batching (MB), which combines both. Experiments in in-domain and out-of-domain settings show that even limited speech consistently improves performance. Notably, MB using only 10% of the target-domain (less than 4 hours) speech achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full dataset, indicating that small amounts of speech provide a strong modality-alignment signal.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

26 of 32 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  7. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  8. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
  9. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  10. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  11. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  12. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
14 more
  1. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  2. 2023Learning from models beyond fine-tuningcited by 48
  3. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  4. 2023SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpuscited by 19
  5. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  6. 2021Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0cited by 12
  7. 2020Cross-Language Transfer Learning, Continuous Learning, and Domain Adaptation for End-to-End Automatic Speech Recognitioncited by 9
  8. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  9. 2022A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Datacited by 7
  10. 2025Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuningcited by 7
  11. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  12. 2025Bridging the Modality Gap: Softly Discretizing Audio Representation for LLM-based Automatic Speech Recognitioncited by 4
  13. 2025Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filteringcited by 3
  14. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2

Built on by

Not cited yet.