mlsanityData to 14 Oct 2026

Self-Supervised Speech Representations are More Phonetic than Semantic

arXiv 2406.08619 ↗First posted 12 Jun 2024cs.CL cs.LG eess.AScited by 71

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. Various analyses suggest that S3Ms encode linguistic properties. In this work, we seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Specifically, we curate a novel dataset of near homophone (phonetically similar) and synonym (semantically similar) word pairs and measure the similarities between S3M word representation pairs. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity. Further, we question whether widely used intent classification datasets such as Fluent Speech Commands and Snips Smartlights are adequate for measuring semantic abilities. Our simple baseline, using only the word identity, surpasses S3M-based models. This corroborates our findings and suggests that high scores on these datasets do not necessarily guarantee the presence of semantic content.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

29 of 42 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  10. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  11. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  12. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
17 more
  1. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  2. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  3. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  4. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  5. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  6. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 36
  7. 2018Spoken Language Understanding on the Edgecited by 27
  8. 2021Speech-language Pre-training for End-to-end Spoken Language Understandingcited by 27
  9. 2022Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Modelscited by 21
  10. 2022Opening the Black Box of wav2vec Feature Encodercited by 18
  11. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  12. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speechcited by 15
  13. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  14. 2018Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrievalcited by 11
  15. 2021Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understandingcited by 11
  16. 2023Understanding Probe Behaviors through Variational Bounds of Mutual Informationcited by 11
  17. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 30; 2026: 31.

71 papers.

Sort:

By area:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  3. 2024Improving Semantic Understanding in Speech Language Models via Brain-tuningcited by 20
  4. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  5. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  6. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  7. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  8. 2026[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmeticcited by 12
  9. 2025Emergent morpho-phonological representations in self-supervised speech modelscited by 11
  10. 2024Estimating the Completeness of Discrete Speech Unitscited by 9
  11. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 9
  12. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
38 more
  1. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
  2. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  3. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  4. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 8
  5. 2024SpeechTaxi: On Multilingual Semantic Speech Classificationcited by 6
  6. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  7. 2025Effective Context in Neural Speech Modelscited by 5
  8. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  9. 2025PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancementcited by 5
  10. 2026Causal Tracing of Audio-Text Fusion in Large Audio Language Modelscited by 5
  11. 2025Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understandingcited by 4
  12. 2025Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Modelscited by 4
  13. 2025On-device Streaming Discrete Speech Unitscited by 4
  14. 2025Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0cited by 4
  15. 2026PRiSM: Benchmarking Phone Realization in Speech Modelscited by 4
  16. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  17. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  18. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semanticallycited by 3
  19. 2025Different Speech Translation Models Encode and Translate Speaker Gender Differentlycited by 3
  20. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  21. 2025Word stress in self-supervised speech models: A cross-linguistic comparisoncited by 3
  22. 2026HuPER: A Human-Inspired Framework for Phonetic Perceptioncited by 3
  23. 2026Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representationscited by 3
  24. 2025Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesiscited by 2
  25. 2025Echoes of Phonetics: Unveiling Relevant Acoustic Cues for ASR via Feature Attributioncited by 2
  26. 2025Entropy-based Coarse and Compressed Semantic Speech Representation Learningcited by 2
  27. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  28. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 2
  29. 2026Rethinking Discrete Speech Representation Tokens for Accent Generationcited by 2
  30. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  31. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 2
  32. 2026Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition modelscited by 2
  33. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  34. 2025Improved Intelligibility of Dysarthric Speech using Conditional Flow Matchingcited by 1
  35. 2025LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenizationcited by 1
  36. 2025Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformerscited by 1
  37. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  38. 2025MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidancescited by 1

50 most cited of 71 All 71 (JSON)