# SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling

arXiv: https://arxiv.org/abs/2510.15851 · HTML: https://mlsanity.com/paper/2510.15851/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks](https://mlsanity.com/paper/2309.07937/) (2023)
- [SLM: Bridge the thin gap between speech and text foundation models](https://mlsanity.com/paper/2310.00230/) (2023)
- [SpeechVerse: A Large-scale Generalizable Audio Language Model](https://mlsanity.com/paper/2405.08295/) (2024)
- [AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs](https://mlsanity.com/paper/2311.06753/) (2023)
- [LLaSM: Large Language and Speech Model](https://mlsanity.com/paper/2308.15930/) (2023)
- [Why Warmup the Learning Rate? Underlying Mechanisms and Improvements](https://mlsanity.com/paper/2406.09405/) (2024)
- [BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5](https://mlsanity.com/paper/2406.19954/) (2024)
- [Zero-shot Slot Filling in the Age of LLMs for Dialogue Systems](https://mlsanity.com/paper/2411.18980/) (2024)

## Built on by (most cited first, 3 of 3)

- [Slot Filling as a Reasoning Task for SpeechLLMs](https://mlsanity.com/paper/2510.19326/) (2025)
- [CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents](https://mlsanity.com/paper/2606.21453/) (2026)
- [Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models](https://mlsanity.com/paper/2608.17102/) (2026)

## Abstract (arXiv)

Slot filling is a crucial subtask in spoken language understanding (SLU), traditionally implemented as a cascade of speech recognition followed by one or more natural language understanding (NLU) components. The recent advent of speech-based large language models (speechLLMs), which integrate speech and textual foundation models, has opened new avenues for achieving speech understanding tasks in a more unified, generative, and instruction-following manner while promising data and compute efficiency with zero-shot abilities, generalizing to unseen slot labels. We address the slot-filling task by creating an empirical upper bound for the task, identifying performance, robustness, and generalization gaps, and proposing improvements to the training data, architecture, and training strategies to narrow the gap with the upper bound result. We show that each of these measures improve performance substantially, while highlighting practical challenges and providing empirical guidance and insights for harnessing these emerging models.