# Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech Data

arXiv: https://arxiv.org/abs/2509.15389 · HTML: https://mlsanity.com/paper/2509.15389/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 22 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs](https://mlsanity.com/paper/2311.06753/) (2023)
- [A Survey on Spoken Language Understanding: Recent Advances and New Frontiers](https://mlsanity.com/paper/2103.03095/) (2021)
- [Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?](https://mlsanity.com/paper/2505.09439/) (2025)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond](https://mlsanity.com/paper/2408.03900/) (2024)
- [Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue](https://mlsanity.com/paper/2409.04927/) (2024)
- [ITALIC: An Italian Intent Classification Dataset](https://mlsanity.com/paper/2306.08502/) (2023)
- [Exploring Finetuned Audio-LLM on Heart Murmur Features](https://mlsanity.com/paper/2501.13884/) (2025)
- [An End-to-End Speech Summarization Using Large Language Model](https://mlsanity.com/paper/2407.02005/) (2024)
- [Direct Speech to Speech Translation: A Review](https://mlsanity.com/paper/2503.04799/) (2025)
- [Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Filling](https://mlsanity.com/paper/2307.07057/) (2023)
- [Fine-Tuning Large Audio-Language Models with LoRA for Precise Temporal Localization of Prolonged Exposure Therapy Elements](https://mlsanity.com/paper/2506.09707/) (2025)
- [Spoken Language Understanding on Unseen Tasks With In-Context Learning](https://mlsanity.com/paper/2505.07731/) (2025)

## Built on by (most cited first, 6 of 6)

- [Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs](https://mlsanity.com/paper/2604.16659/) (2026)
- [Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades](https://mlsanity.com/paper/2605.17443/) (2026)
- [CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models](https://mlsanity.com/paper/2602.07077/) (2026)
- [CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents](https://mlsanity.com/paper/2606.21453/) (2026)
- [Efficiently Adapting Spoken Language Models for the Singaporean Context](https://mlsanity.com/paper/2607.10092/) (2026)
- [Robust Summarization of Doctor-Patient Conversations: TalTech Systems for the Beyond Transcription Challenge](https://mlsanity.com/paper/2607.17230/) (2026)

## Abstract (arXiv)

Large Audio Language Models (LALMs) have emerged as powerful tools for speech-related tasks but remain underexplored for fine-tuning, especially with limited speech data. To bridge this gap, we systematically examine how different fine-tuning schemes including text-only, direct mixing, and curriculum learning affect spoken language understanding (SLU), focusing on scenarios where text-label pairs are abundant while paired speech-label data are limited. Results show that LALMs already achieve competitive performance with text-only fine-tuning, highlighting their strong generalization ability. Adding even small amounts of speech data (2-5%) yields substantial further gains, with curriculum learning particularly effective under scarce data. In cross-lingual SLU, combining source-language speech data with target-language text and minimal target-language speech data enables effective adaptation. Overall, this study provides practical insights into the LALM fine-tuning under realistic data constraints.