mlsanityData to 7 Oct 2026

Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering

arXiv 2305.11072 ↗First posted 18 May 2023cs.CL eess.AScited by 21

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech representation models have succeeded in various tasks, but improving them for content-related problems using unlabeled data is challenging. We propose speaker-invariant clustering (Spin), a novel self-supervised learning method that clusters speech representations and performs swapped prediction between the original and speaker-perturbed utterances. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU. Spin improves pre-trained networks and outperforms prior methods in speech recognition and acoustic unit discovery.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

32 of 39 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  12. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
20 more
  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  3. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  4. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  5. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  6. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  7. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  8. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  9. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  10. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  11. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  12. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  13. 2022Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptationcited by 16
  14. 2022Speech Pre-training with Acoustic Piececited by 14
  15. 2020Unsupervised Learning of Disentangled Speech Content and Style Representationcited by 13
  16. 2021Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Modelscited by 9
  17. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  18. 2022Content-Context Factorized Representations for Automated Speech Recognitioncited by 8
  19. 2021Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclawcited by 7
  20. 2022Towards Disentangled Speech Representationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 5; 2026: 3.

21 papers.

Sort:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  3. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  4. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  5. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  6. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  7. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  8. 2025fastabx: A library for efficient computation of ABX discriminabilitycited by 7
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  10. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  11. 2024Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERTcited by 4
  12. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
9 more
  1. 2024SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representationscited by 3
  2. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  3. 2024LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Taskscited by 2
  4. 2024Towards the Next Frontier in Speech Representation Learning Using Disentanglementcited by 1
  5. 2023HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Modelcited by 0
  6. 2024JOOCI: a Framework for Learning Comprehensive Speech Representationscited by 0
  7. 2026Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representationscited by 0
  8. 2026Frontend Token Enhancement for Token-Based Speech Recognitioncited by 0
  9. 2026SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Trainingcited by 0