mlsanityData to 14 Oct 2026

A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding

arXiv 2211.05869 ↗First posted 10 Nov 2022cs.CL cs.SD eess.AScited by 18

Not on the map. Why?

Abstract (arXiv)

Collecting sufficient labeled data for spoken language understanding (SLU) is expensive and time-consuming. Recent studies achieved promising results by using pre-trained models in low-resource scenarios. Inspired by this, we aim to ask: which (if any) pre-training strategies can improve performance across SLU benchmarks? To answer this question, we employ four types of pre-trained models and their combinations for SLU. We leverage self-supervised speech and language models (LM) pre-trained on large quantities of unpaired data to extract strong speech and text representations. We also explore using supervised models pre-trained on larger external automatic speech recognition (ASR) or SLU corpora. We conduct extensive experiments on the SLU Evaluation (SLUE) benchmark and observe self-supervised pre-trained models to be more powerful, with pre-trained LM and speech models being most beneficial for the Sentiment Analysis and Named Entity Recognition task, respectively.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

41 of 53 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2019When Does Label Smoothing Help?cited by 988
29 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  4. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  5. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  6. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  7. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  8. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  9. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2018Towards end-to-end spoken language understandingcited by 92
  12. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  13. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  14. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  15. 2019Two-Pass End-to-End Speech Recognitioncited by 64
  16. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
  17. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 62
  18. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
  19. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  20. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  21. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  22. 2018Spoken Language Understanding on the Edgecited by 27
  23. 2020Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understandingcited by 23
  24. 2019Gunrock: A Social Bot for Complex and Engaging Long Conversationscited by 14
  25. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  26. 2021Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understandingcited by 11
  27. 2020Semantic Complexity in End-to-End Spoken Language Understandingcited by 10
  28. 2018Lexico-acoustic Neural-based Models for Dialog Act Classificationcited by 8
  29. 2022Two-Pass Low Latency End-to-End Spoken Language Understandingcited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 10; 2024: 3; 2025: 4.

18 papers.

Sort:

By area:

  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  3. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  4. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  5. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  6. 2024Exploring the Capability of Mamba in Speech Applicationscited by 20
  7. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 18
  8. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 14
  9. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  10. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 5
  11. 2023Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understandingcited by 3
  12. 2023Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understandingcited by 2
6 more
  1. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  2. 2023Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understandingcited by 1
  3. 2025K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Functioncited by 1
  4. 2023Continual Contrastive Spoken Language Understandingcited by 0
  5. 2023HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Modelcited by 0
  6. 2025Joint Automatic Speech Recognition And Structure Learning For Better Speech Understandingcited by 0