Investigation of Speaker Representation for Target-Speaker Speech Processing
Not on the map. Why?
Abstract (arXiv)
Target-speaker speech processing (TS) tasks, such as target-speaker automatic speech recognition (TS-ASR), target speech extraction (TSE), and personal voice activity detection (p-VAD), are important for extracting information about a desired speaker's speech even when it is corrupted by interfering speakers. While most studies have focused on training schemes or system architectures for each specific task, the auxiliary network for embedding target-speaker cues has not been investigated comprehensively in a unified cross-task evaluation. Therefore, this paper aims to address a fundamental question: what is the preferred speaker embedding for TS tasks? To this end, for the TS-ASR, TSE, and p-VAD tasks, we compare pre-trained speaker encoders (i.e., self-supervised or speaker recognition models) that compute speaker embeddings from pre-recorded enrollment speech of the target speaker with ideal speaker embeddings derived directly from the target speaker's identity in the form of a one-hot vector. To further understand the properties of ideal speaker embedding, we optimize it using a gradient-based approach to improve performance on the TS task. Our analysis reveals that speaker verification performance is somewhat unrelated to TS task performances, the one-hot vector outperforms enrollment-based ones, and the optimal embedding depends on the input mixture.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 15 Oct 2024.
Builds on
43 of 57 references.
References · 43
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2016Adversarial examples in the physical worldcited by 1,762
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
31 more
- 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
- 2018SDR - half-baked or well done?cited by 420
- 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
- 2015Understanding Neural Networks Through Deep Visualizationcited by 332
- 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
- 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
- 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
- 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
- 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
- 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 67
- 2020Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learningcited by 29
- 2019Personal VAD: Speaker-Conditioned Voice Activity Detectioncited by 24
- 2020Improving speaker discrimination of target speech extraction with time-domain SpeakerBeamcited by 20
- 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
- 2022An attention-based backend allowing efficient fine-tuning of transformer models for speaker verificationcited by 14
- 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
- 2019Self-supervised speaker embeddingscited by 12
- 2022Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognitioncited by 12
- 2022Pushing the limits of self-supervised speaker verification using regularized distillation frameworkcited by 11
- 2022Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learningcited by 10
- 2023Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audiocited by 10
- 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 10
- 2022C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verificationcited by 9
- 2023VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challengecited by 8
- 2022A comprehensive study on self-supervised distillation for speaker representation learningcited by 6
- 2024Probing Self-supervised Learning Models with Target Speech Extractioncited by 5
- 2022Non-Contrastive Self-Supervised Learning of Utterance-Level Speech Representationscited by 3
- 2022Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separationcited by 3
- 2023Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Datacited by 3
- 2023The second multi-channel multi-party meeting transcription challenge (M2MeT) 2.0): A benchmark for speaker-attributed ASRcited by 3
- 2023Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditionscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.