# Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding

arXiv: https://arxiv.org/abs/2406.15209 · HTML: https://mlsanity.com/paper/2406.15209/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [WhisperNER: Unified Open Named Entity and Speech Recognition](https://mlsanity.com/paper/2409.08107/) (2024)
- [SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings](https://mlsanity.com/paper/2509.04473/) (2025)
- [Refining Transcripts With TV Subtitles by Prompt-Based Weakly Supervised Training of ASR](https://mlsanity.com/paper/2509.04491/) (2025)
- [SFL-MTSC: Leveraging Semantic Frame-Level Multi-Task Self-Consistency for Robust Multi-Intent Spoken Language Understanding](https://mlsanity.com/paper/2606.25552/) (2026)

## Abstract (arXiv)

Zero-shot spoken language understanding (SLU) enables systems to comprehend user utterances in new domains without prior exposure to training data. Recent studies often rely on large language models (LLMs), leading to excessive footprints and complexity. This paper proposes the use of Whisper, a standalone speech processing model, for zero-shot end-to-end (E2E) SLU. To handle unseen semantic labels, SLU tasks are integrated into a question-answering (QA) framework, which prompts the Whisper decoder for semantics deduction. The system is efficiently trained with prefix-tuning, optimising a minimal set of parameters rather than the entire Whisper model. We show that the proposed system achieves a 40.7% absolute gain for slot filling (SLU-F1) on SLURP compared to a recently introduced zero-shot benchmark. Furthermore, it performs comparably to a Whisper-GPT-2 modular system under both in-corpus and cross-corpus evaluation settings, but with a relative 34.8% reduction in model parameters.