# WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding

arXiv: https://arxiv.org/abs/2408.16423 · HTML: https://mlsanity.com/paper/2408.16423/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2024.

## Built on by (most cited first, 8 of 8)

- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward](https://mlsanity.com/paper/2411.03866/) (2024)
- [Multi-Document Financial Question Answering using LLMs](https://mlsanity.com/paper/2411.07264/) (2024)
- [ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models](https://mlsanity.com/paper/2507.20091/) (2025)
- [MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark](https://mlsanity.com/paper/2512.01603/) (2025)
- [PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark](https://mlsanity.com/paper/2603.14456/) (2026)
- [Spooftral: Can Voxtral Audio-Language Model Detect Speech Spoofing?](https://mlsanity.com/paper/2609.28713/) (2026)

## Abstract (arXiv)

Speech large language models (speech-LLMs) integrate speech and text-based foundation models to provide a unified framework for handling a wide range of downstream tasks. In this paper, we introduce WHISMA, a speech-LLM tailored for spoken language understanding (SLU) that demonstrates robust performance in various zero-shot settings. WHISMA combines the speech encoder from Whisper with the Llama-3 LLM, and is fine-tuned in a parameter-efficient manner on a comprehensive collection of SLU-related datasets. Our experiments show that WHISMA significantly improves the zero-shot slot filling performance on the SLURP benchmark, achieving a relative gain of 26.6% compared to the current state-of-the-art model. Furthermore, to evaluate WHISMA's generalisation capabilities to unseen domains, we develop a new task-agnostic benchmark named SLU-GLUE. The evaluation results indicate that WHISMA outperforms an existing speech-LLM (Qwen-Audio) with a relative gain of 33.0%.