mlsanityData to 14 Oct 2026

Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning

arXiv 2506.05671 ↗First posted 6 Jun 2025eess.AS cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent advances in automatic speech recognition (ASR) have combined speech encoders with large language models (LLMs) through projection, forming Speech LLMs with strong performance. However, adapting them to new domains remains challenging, especially in low-resource settings where paired speech-text data is scarce. We propose a text-only fine-tuning strategy for Speech LLMs using unpaired target-domain text without requiring additional audio. To preserve speech-text alignment, we introduce a real-time evaluation mechanism during fine-tuning. This enables effective domain adaptation while maintaining source-domain performance. Experiments on LibriSpeech, SlideSpeech, and Medical datasets show that our method achieves competitive recognition performance, with minimal degradation compared to full audio-text fine-tuning. It also improves generalization to new domains without catastrophic forgetting, highlighting the potential of text-only fine-tuning for low-resource domain adaptation of ASR.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

26 of 42 references.

References · 26

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  4. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  5. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  6. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  7. 2018Deep context: end-to-end contextual speech recognitioncited by 137
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  9. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 88
  10. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  11. 2024A Survey on Speech Large Language Models for Understandingcited by 71
  12. 2024Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognitioncited by 63
14 more
  1. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  2. 2025FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integrationcited by 46
  3. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  4. 2023Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Networkcited by 31
  5. 2023Revisiting Large Language Models as Zero-shot Relation Extractorscited by 29
  6. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  7. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  8. 2023SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpuscited by 19
  9. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  10. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  11. 2025OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academiacited by 13
  12. 2023Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducercited by 12
  13. 2022Improving Mandarin Speech Recogntion with Block-augmented Transformercited by 6
  14. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  2. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2
  3. 2026On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASRcited by 2
  4. 2025WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformerscited by 0
  5. 2026Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASRcited by 0
  6. 2026TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMscited by 0
  7. 2026CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agentscited by 0