mlsanityData to 7 Oct 2026

Disentangling Voice and Content with Self-Supervision for Speaker Recognition

arXiv 2310.01128 ↗First posted 2 Oct 2023eess.AS cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

For speaker recognition, it is difficult to extract an accurate speaker representation from speech because of its mixture of speaker traits and content. This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is realized with the use of three Gaussian inference layers, each consisting of a learnable transition model that extracts distinct speech components. Notably, a strengthened transition model is specifically designed to model complex speech dynamics. We also propose a self-supervision method to dynamically disentangle content without the use of labels other than speaker identities. The efficacy of the proposed framework is validated via experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF, respectively. Since neither additional model training nor data is specifically needed, it is easily applicable in practical use.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 61 of its 104 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

61 of 104 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  12. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
38 more
  1. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  2. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  3. 2019Similarity-Preserving Knowledge Distillationcited by 507
  4. 2023A Cookbook of Self-Supervised Learningcited by 248
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  7. 2019ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detectioncited by 173
  8. 2020In defence of metric learning for speaker recognitioncited by 123
  9. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  10. 2018Attentive Statistics Pooling for Deep Speaker Embeddingcited by 108
  11. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  12. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  13. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  14. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  15. 2019Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spacescited by 64
  16. 2020Speaker Recognition Based on Deep Learning: An Overviewcited by 59
  17. 2015End-to-End Text-Dependent Speaker Verificationcited by 49
  18. 2019Utterance-level Aggregation For Speaker Recognition In The Wildcited by 49
  19. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  20. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 30
  21. 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 28
  22. 2018Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representationscited by 25
  23. 2018Speaker-Invariant Training via Adversarial Learningcited by 23
  24. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  25. 2019Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?cited by 22
  26. 2020The ins and outs of speaker recognition: lessons from VoxSRC 2020cited by 21
  27. 2021Selective Listening by Synchronizing Speech with Lipscited by 15
  28. 2019Speaker-invariant Affective Representation Learning via Adversarial Trainingcited by 14
  29. 2022MFA: TDNN with Multi-scale Frequency-channel Attention for Text-independent Speaker Verification with Short Utterancescited by 12
  30. 2020Probabilistic embeddings for speaker diarizationcited by 11
  31. 2021Integrating Frequency Translational Invariance in TDNNs and Frequency Positional Information in 2D ResNets to Enhance Speaker Verificationcited by 11
  32. 2021Self-supervised Speaker Recognition with Loss-gated Learningcited by 10
  33. 2022Frequency and Multi-Scale Selective Kernel Attention for Speaker Verificationcited by 10
  34. 2018Speaker Embedding Extraction with Phonetic Informationcited by 9
  35. 2022VoxSRC 2021: The Third VoxCeleb Speaker Recognition Challengecited by 9
  36. 2021Attention Back-end for Automatic Speaker Verification with Multiple Enrollment Utterancescited by 8
  37. 2021Xi-Vector Embedding for Speaker Recognitioncited by 8
  38. 2020An iterative framework for self-supervised deep speaker representation learningcited by 7

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 3.

10 papers.

  1. 2024Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detectioncited by 17
  2. 2025Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofingcited by 17
  3. 2024How Do Neural Spoofing Countermeasures Detect Partially Spoofed Audio?cited by 10
  4. 2024Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024cited by 7
  5. 2024Towards Quantifying and Reducing Language Mismatch Effects in Cross-Lingual Speech Anti-Spoofingcited by 7
  6. 2026Scores Know Bobs Voice: Speaker Impersonation Attackcited by 2
  7. 2025Interpolating Speaker Identities in Embedding Space for Data Expansioncited by 1
  8. 2024Cosine Scoring with Uncertainty for Neural Speaker Embeddingcited by 0
  9. 2026Robust Generative Audio Quality Assessment: Disentangling Quality from Spurious Correlationscited by 0
  10. 2026Listening Alone, Understanding Together: Collaborative Context Recovery for Privacy-Aware AIcited by 0