mlsanityData to 14 Oct 2026

What Do Self-Supervised Speech Models Know About Words?

arXiv 2307.00162 ↗First posted 30 Jun 2023cs.CL cs.LG eess.AScited by 64

Not on the map. Why?

Abstract (arXiv)

Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete picture of what is learned during pre-training. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information, but we still lack a proper understanding of knowledge encoded at the word level and beyond. In this work, we use lightweight analysis methods to study segment-level linguistic properties -- word identity, boundaries, pronunciation, syntactic features, and semantic features -- encoded in S3Ms. We present a comparative study of layer-wise representations from ten S3Ms and find that (i) the frame-level representations within each word segment are not all equally informative, and (ii) the pre-training objective and model size heavily influence the accessibility and distribution of linguistic information across layers. We also find that on several tasks -- word discrimination, word segmentation, and semantic sentence similarity -- S3Ms trained with visual grounding outperform their speech-only counterparts. Finally, our task-based analyses demonstrate improved performance on word segmentation and acoustic word discrimination while using simpler methods than prior work.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 81 of its 113 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

81 of 113 references.

References · 81

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2019Designing and Interpreting Probes with Control Taskscited by 640
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
38 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  2. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  4. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  5. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  9. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  12. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  13. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  14. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  15. 2021Unsupervised Speech Recognitioncited by 148
  16. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  17. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  18. 2022Generative Spoken Dialogue Language Modelingcited by 126
  19. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  20. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  21. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  22. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  23. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  24. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  25. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  26. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  27. 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
  28. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  29. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  30. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  31. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  32. 2022Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learningcited by 29
  33. 2015Deep convolutional acoustic word embeddings using word-pair side informationcited by 27
  34. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  35. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  36. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  37. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  38. 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 23; 2026: 23.

64 papers.

Sort:

By area:

  1. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  2. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  3. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  4. 2024Encoding of lexical tone in self-supervised models of spoken languagecited by 23
  5. 2024Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0cited by 19
  6. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  7. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  8. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  9. 2026[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmeticcited by 12
  10. 2023Understanding Probe Behaviors through Variational Bounds of Mutual Informationcited by 11
  11. 2025Emergent morpho-phonological representations in self-supervised speech modelscited by 11
  12. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 11
38 more
  1. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  2. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 8
  3. 2024Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programmingcited by 7
  4. 2025fastabx: A library for efficient computation of ABX discriminabilitycited by 7
  5. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  6. 2024Exploring bat song syllable representations in self-supervised audio encoderscited by 5
  7. 2025Effective Context in Neural Speech Modelscited by 5
  8. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  9. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  10. 2024Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERTcited by 4
  11. 2025What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-trainingcited by 4
  12. 2025Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?cited by 4
  13. 2025Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0cited by 4
  14. 2025Unsupervised lexicon learning from speech is limited by representations rather than clusteringcited by 4
  15. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  16. 2024Investigating the 'Autoencoder Behavior' in Speech Self-Supervised Models: a focus on HuBERT's Pretrainingcited by 3
  17. 2024Perception of Phonological Assimilation by Neural Speech Recognition Modelscited by 3
  18. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  19. 2026Mind the Shift: Using Delta SSL Embeddings to Enhance Child ASRcited by 3
  20. 2024Convexity-based Pruning of Speech Representation Modelscited by 2
  21. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  22. 2025Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Modelscited by 2
  23. 2026Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networkscited by 2
  24. 2023Few-Shot Spoken Language Understanding via Joint Speech-Text Modelscited by 1
  25. 2024Using Shapley interactions to understand how models use structurecited by 1
  26. 2024Towards Unsupervised Speech Recognition Without Pronunciation Modelscited by 1
  27. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  28. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  29. 2026Tracking the emergence of linguistic structure in self-supervised models learning from speechcited by 1
  30. 2026Revisiting Lexicon Evaluation in Unsupervised Word Discoverycited by 1
  31. 2026Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluationcited by 1
  32. 2026A Native-Reference Coordinate Geometry for L2 Pronunciation Deviation Using Self-Supervised Speech Modelscited by 1
  33. 2024An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutionscited by 0
  34. 2025Emphasis Sensitivity in Speech Representationscited by 0
  35. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  36. 2025Large language models have learned to use languagecited by 0
  37. 2025Large language models are not about natural languagecited by 0
  38. 2025Linguists should learn to love speech-based deep learning modelscited by 0

50 most cited of 64 All 64 (JSON)