# Guided Speaker Embedding

arXiv: https://arxiv.org/abs/2410.12182 · HTML: https://mlsanity.com/paper/2410.12182/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification](https://mlsanity.com/paper/2010.11255/) (2020)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech](https://mlsanity.com/paper/2105.09040/) (2021)
- [Streaming end-to-end multi-talker speech recognition](https://mlsanity.com/paper/2011.13148/) (2020)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [Multi-turn RNN-T for streaming recognition of multi-party speech](https://mlsanity.com/paper/2112.10200/) (2021)
- [Streaming Multi-talker Speech Recognition with Joint Speaker Identification](https://mlsanity.com/paper/2104.02109/) (2021)
- [Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Models](https://mlsanity.com/paper/1909.08103/) (2019)
- [Endpoint Detection for Streaming End-to-End Multi-talker ASR](https://mlsanity.com/paper/2201.09979/) (2022)
- [Speaker activity driven neural speech extraction](https://mlsanity.com/paper/2101.05516/) (2021)
- [Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecture](https://mlsanity.com/paper/2309.09180/) (2023)
- [NTT speaker diarization system for CHiME-7: multi-domain, multi-microphone End-to-end and vector clustering diarization](https://mlsanity.com/paper/2309.12656/) (2023)
- [Alignment-Free Training for Transducer-based Multi-Talker ASR](https://mlsanity.com/paper/2409.20301/) (2024)
- [Multi-channel Conversational Speaker Separation via Neural Diarization](https://mlsanity.com/paper/2311.08630/) (2023)
- [MIRNet: Learning multiple identities representations in overlapped speech](https://mlsanity.com/paper/2008.01698/) (2020)
- [A Teacher-Student approach for extracting informative speaker embeddings from speech mixtures](https://mlsanity.com/paper/2306.00634/) (2023)
- [Recursive Attentive Pooling for Extracting Speaker Embeddings from Multi-Speaker Recordings](https://mlsanity.com/paper/2408.17142/) (2024)

## Built on by (most cited first, 1 of 1)

- [Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation](https://mlsanity.com/paper/2411.13849/) (2024)

## Abstract (arXiv)

This paper proposes a guided speaker embedding extraction system, which extracts speaker embeddings of the target speaker using speech activities of target and interference speakers as clues. Several methods for long-form overlapped multi-speaker audio processing are typically two-staged: i) segment-level processing and ii) inter-segment speaker matching. Speaker embeddings are often used for the latter purpose. Typical speaker embedding extraction approaches only use single-speaker intervals to avoid corrupting the embeddings with speech from interference speakers. However, this often makes speaker embeddings impossible to extract because sufficiently long non-overlapping intervals are not always available. In this paper, we propose using speaker activities as clues to extract the embedding of the speaker-of-interest directly from overlapping speech. Specifically, we concatenate the activity of target and non-target speakers to acoustic features before being fed to the model. We also condition the attention weights used for pooling so that the attention weights of the intervals in which the target speaker is inactive are zero. The effectiveness of the proposed method is demonstrated in speaker verification and speaker diarization.