# SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech

arXiv: https://arxiv.org/abs/2111.10367 · HTML: https://mlsanity.com/paper/2111.10367/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://mlsanity.com/paper/2109.06870/) (2021)
- [Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition](https://mlsanity.com/paper/1912.01679/) (2019)
- [Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation](https://mlsanity.com/paper/2005.08575/) (2020)
- [Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction](https://mlsanity.com/paper/2001.10603/) (2020)
- [End-to-end Named Entity Recognition from English Speech](https://mlsanity.com/paper/2005.11184/) (2020)
- [Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining](https://mlsanity.com/paper/2010.13826/) (2020)
- [Improving Transformer-based Speech Recognition Using Unsupervised Pre-training](https://mlsanity.com/paper/1910.09932/) (2019)
- [Learning Speaker Representations with Mutual Information](https://mlsanity.com/paper/1812.00271/) (2018)
- [Augmentation adversarial training for self-supervised speaker recognition](https://mlsanity.com/paper/2007.12085/) (2020)
- [Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models](https://mlsanity.com/paper/1911.09762/) (2019)
- [Contrastive Unsupervised Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2102.06357/) (2021)
- [Audio De-identification: A New Entity Recognition Task](https://mlsanity.com/paper/1903.07037/) (2019)
- [ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understanding](https://mlsanity.com/paper/2108.13048/) (2021)

## Built on by (most cited first, 30 of 72)

- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech](https://mlsanity.com/paper/2309.09510/) (2023)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing](https://mlsanity.com/paper/2309.00916/) (2023)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023)
- [Toward Joint Language Modeling for Speech Units and Text](https://mlsanity.com/paper/2310.08715/) (2023)
- [Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](https://mlsanity.com/paper/2205.01086/) (2022)
- [AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs](https://mlsanity.com/paper/2510.07293/) (2025)
- [Exploring the Capability of Mamba in Speech Applications](https://mlsanity.com/paper/2406.16808/) (2024)
- [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)
- [HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts](https://mlsanity.com/paper/2404.17507/) (2024)
- [Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue](https://mlsanity.com/paper/2409.04927/) (2024)
- [A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks](https://mlsanity.com/paper/2305.11073/) (2023)
- [MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols](https://mlsanity.com/paper/2508.18240/) (2025)
- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models](https://mlsanity.com/paper/2210.15734/) (2022)
- [DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding](https://mlsanity.com/paper/2406.09345/) (2024)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?](https://mlsanity.com/paper/2306.08374/) (2023)
- [On the Evaluation of Speech Foundation Models for Spoken Language Understanding](https://mlsanity.com/paper/2406.10083/) (2024)
- [Context-aware Fine-tuning of Self-supervised Speech Models](https://mlsanity.com/paper/2212.08542/) (2022)
- [Contextual Biasing of Named-Entities with Large Language Models](https://mlsanity.com/paper/2309.00723/) (2023)
- [Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks](https://mlsanity.com/paper/2502.13645/) (2025)

## Abstract (arXiv)

Progress in speech processing has been facilitated by shared datasets and benchmarks. Historically these have focused on automatic speech recognition (ASR), speaker identification, or other lower-level tasks. Interest has been growing in higher-level spoken language understanding tasks, including using end-to-end models, but there are fewer annotated datasets for such tasks. At the same time, recent work shows the possibility of pre-training generic representations and then fine-tuning for several tasks using relatively little labeled data. We propose to create a suite of benchmark tasks for Spoken Language Understanding Evaluation (SLUE) consisting of limited-size labeled training sets and corresponding evaluation sets. This resource would allow the research community to track progress, evaluate pre-trained representations for higher-level tasks, and study open questions such as the utility of pipeline versus end-to-end approaches. We present the first phase of the SLUE benchmark suite, consisting of named entity recognition, sentiment analysis, and ASR on the corresponding datasets. We focus on naturally produced (not read or synthesized) speech, and freely available datasets. We provide new transcriptions and annotations on subsets of the VoxCeleb and VoxPopuli datasets, evaluation metrics and results for baseline models, and an open-source toolkit to reproduce the baselines and evaluate new models.