# SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks

arXiv: https://arxiv.org/abs/2212.10525 · HTML: https://mlsanity.com/paper/2212.10525/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 86 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 30 of 56)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)

## Built on by (most cited first, 30 of 35)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [VoiceAgentBench: Are Voice Assistants ready for agentic tasks?](https://mlsanity.com/paper/2510.07978/) (2025)
- [WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models](https://mlsanity.com/paper/2502.14727/) (2025)
- [HeySQuAD: A Spoken Question Answering Dataset](https://mlsanity.com/paper/2304.13689/) (2023)
- [Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue](https://mlsanity.com/paper/2409.04927/) (2024)
- [SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answering](https://mlsanity.com/paper/2401.13463/) (2024)
- [DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding](https://mlsanity.com/paper/2406.09345/) (2024)
- [On the Evaluation of Speech Foundation Models for Spoken Language Understanding](https://mlsanity.com/paper/2406.10083/) (2024)
- [MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models](https://mlsanity.com/paper/2412.09818/) (2024)
- [Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume](https://mlsanity.com/paper/2602.24195/) (2026)
- [Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks](https://mlsanity.com/paper/2502.13645/) (2025)
- [The Role of Prosody in Spoken Question Answering](https://mlsanity.com/paper/2502.05389/) (2025)
- [AugSumm: towards generalizable speech summarization using synthetic labels from large language model](https://mlsanity.com/paper/2401.06806/) (2024)
- [Interface Design for Self-Supervised Speech Models](https://mlsanity.com/paper/2406.12209/) (2024)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [BASS: Block-wise Adaptation for Speech Summarization](https://mlsanity.com/paper/2307.08217/) (2023)
- [An End-to-End Speech Summarization Using Large Language Model](https://mlsanity.com/paper/2407.02005/) (2024)
- [AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence](https://mlsanity.com/paper/2512.10624/) (2025)
- [Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?](https://mlsanity.com/paper/2408.07277/) (2024)
- [WhisperNER: Unified Open Named Entity and Speech Recognition](https://mlsanity.com/paper/2409.08107/) (2024)
- [NUTSHELL: A Dataset for Abstract Generation from Scientific Talks](https://mlsanity.com/paper/2502.16942/) (2025)
- [What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels](https://mlsanity.com/paper/2512.16832/) (2025)
- [Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understanding](https://mlsanity.com/paper/2307.11005/) (2023)
- [MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders](https://mlsanity.com/paper/2409.06635/) (2024)
- [Spoken Language Understanding on Unseen Tasks With In-Context Learning](https://mlsanity.com/paper/2505.07731/) (2025)
- [End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering](https://mlsanity.com/paper/2511.09282/) (2025)
- [Factual Consistency Oriented Speech Recognition](https://mlsanity.com/paper/2302.12369/) (2023)
- [Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model](https://mlsanity.com/paper/2402.05819/) (2024)
- [Medical Spoken Named Entity Recognition](https://mlsanity.com/paper/2406.13337/) (2024)

## Abstract (arXiv)

Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In particular, there are not nearly as many SLU task benchmarks, and many of the existing ones use data that is not freely available to all researchers. Recent work has begun to introduce such benchmark datasets for several tasks. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. We follow the blueprint of the Spoken Language Understanding Evaluation (SLUE) benchmark suite. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will be publishing for each task (i) annotations for a relatively small fine-tuning set, (ii) annotated development and test sets, and (iii) baseline models for easy reproducibility and comparisons. In this work, we present the details of data collection and annotation and the performance of the baseline models. We also perform sensitivity analysis of pipeline models' performance (speech recognizer + text model) to the speech recognition accuracy, using more than 20 state-of-the-art speech recognition models.