mlsanityData to 14 Oct 2026

SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT

arXiv 2310.10803 ↗First posted 16 Oct 2023cs.CL eess.AScited by 13

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing. Yet, the discovered units often remain in phonetic space and the units beyond phonemes are largely underexplored. Here, we demonstrate that a syllabic organization emerges in learning sentence-level representation of speech. In particular, we adopt "self-distillation" objective to fine-tune the pretrained HuBERT with an aggregator token that summarizes the entire sentence. Without any supervision, the resulting model draws definite boundaries in speech, and the representations across frames exhibit salient syllabic structures. We demonstrate that this emergent structure largely corresponds to the ground truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech. When compared to previous models, our model outperforms in both unsupervised syllable discovery and learning sentence-level representation. Together, we demonstrate that the self-distillation of HuBERT gives rise to syllabic organization without relying on external labels or modalities, and potentially provides novel data-driven units for spoken language modeling.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 23 of its 31 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

23 of 31 references.

References · 23

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  11. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  12. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
11 more
  1. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  2. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  3. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  4. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  5. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  6. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  7. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  8. 2017An embedded segmental K-means model for unsupervised segmentation and clustering of speechcited by 31
  9. 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 31
  10. 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
  11. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speechcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  3. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  4. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  5. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  6. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  7. 2024Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERTcited by 4
  8. 2025Towards Unsupervised Speech Recognition at the Syllable-Levelcited by 2
  9. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  10. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  11. 2026findsylls: A Language-Agnostic Toolkit for Syllable-Level Speech Tokenization and Embeddingcited by 0
  12. 2026Recovering the Zipfian Distribution in Unsupervised Term Discoverycited by 0
1 more
  1. 2026Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
  2. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
  3. 2022Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAE
  4. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  5. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
  6. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
  7. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
  8. 2024Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective