# Recursive Attentive Pooling for Extracting Speaker Embeddings from Multi-Speaker Recordings arXiv: https://arxiv.org/abs/2408.17142 ยท HTML: https://mlsanity.com/paper/2408.17142/ ## Key facts - Cited by 2 papers in the map (data through 19 Oct 2026). - Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. - Builds on 42 of its 58 references that are in the map. - First posted on arXiv 30 Aug 2024. ## Placed in - Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/) ## Builds on (most cited first, 30 of 42) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017) - [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017) - [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020) - [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015) - [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017) - [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015) - [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018) - [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020) - [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021) - [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019) - [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020) - [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020) - [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018) - [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016) - [Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap](https://mlsanity.com/paper/2003.02405/) (2020) - [Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech](https://mlsanity.com/paper/2105.09040/) (2021) - [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020) - [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023) - [Encoder-Decoder Based Attractors for End-to-End Neural Diarization](https://mlsanity.com/paper/2106.10654/) (2021) - [ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models](https://mlsanity.com/paper/2401.17230/) (2024) - [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019) - [Personal VAD: Speaker-Conditioned Voice Activity Detection](https://mlsanity.com/paper/1908.04284/) (2019) - [Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.11458/) (2020) - [From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.00890/) (2022) - [Self Multi-Head Attention for Speaker Recognition](https://mlsanity.com/paper/1906.09890/) (2019) - [End-to-End Speaker Diarization as Post-Processing](https://mlsanity.com/paper/2012.10055/) (2020) - [Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker](https://mlsanity.com/paper/2108.03342/) (2021) - [Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition](https://mlsanity.com/paper/2201.03881/) (2022) ## Built on by (most cited first, 2 of 2) - [Pretraining Multi-Speaker Identification for Neural Speaker Diarization](https://mlsanity.com/paper/2505.24545/) (2025) - [Guided Speaker Embedding](https://mlsanity.com/paper/2410.12182/) (2024) ## Abstract (arXiv) This paper proposes a method for extracting speaker embedding for each speaker from a variable-length recording containing multiple speakers. Speaker embeddings are crucial not only for speaker recognition but also for various multi-speaker speech applications such as speaker diarization and target-speaker speech processing. Despite the challenges of obtaining a single speaker's speech without pre-registration in multi-speaker scenarios, most studies on speaker embedding extraction focus on extracting embeddings only from single-speaker recordings. Some methods have been proposed for extracting speaker embeddings directly from multi-speaker recordings, but they typically require preparing a model for each possible number of speakers or involve complicated training procedures. The proposed method computes the embeddings of multiple speakers by focusing on different parts of the frame-wise embeddings extracted from the input multi-speaker audio. This is achieved by recursively computing attention weights for pooling the frame-wise embeddings. Additionally, we propose using the calculated attention weights to estimate the number of speakers in the recording, which allows the same model to be applied to various numbers of speakers. Experimental evaluations demonstrate the effectiveness of the proposed method in speaker verification and diarization tasks.