mlsanityData to 7 Oct 2026

SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?

arXiv 2306.08374 ↗First posted 14 Jun 2023cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) for speech representation has been successfully applied in various downstream tasks, such as speech and speaker recognition. More recently, speech SSL models have also been shown to be beneficial in advancing spoken language understanding tasks, implying that the SSL models have the potential to learn not only acoustic but also linguistic information. In this paper, we aim to clarify if speech SSL techniques can well capture linguistic knowledge. For this purpose, we introduce SpeechGLUE, a speech version of the General Language Understanding Evaluation (GLUE) benchmark. Since GLUE comprises a variety of natural language understanding tasks, SpeechGLUE can elucidate the degree of linguistic ability of speech SSL models. Experiments demonstrate that speech SSL models, although inferior to text-based SSL models, perform better than baselines, suggesting that they can acquire a certain amount of general linguistic knowledge from just unlabeled speech data.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

36 of 46 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2018Neural Network Acceptability Judgmentscited by 952
24 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  5. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  8. 2021A Survey on Neural Speech Synthesiscited by 182
  9. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  10. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  13. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  14. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  15. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  16. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  17. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  18. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  19. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  20. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  21. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  22. 2022Speech Pre-training with Acoustic Piececited by 14
  23. 2021ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understandingcited by 9
  24. 2023Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 2; 2026: 1.

9 papers.

  1. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  2. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  3. 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 10
  4. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  5. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  6. 2025PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancementcited by 5
  7. 2024Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Trainingcited by 3
  8. 2024Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillationcited by 3
  9. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'acited by 1