# Multi-Level Speaker Representation for Target Speaker Extraction

arXiv: https://arxiv.org/abs/2410.16059 · HTML: https://mlsanity.com/paper/2410.16059/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2024.

## Built on by (most cited first, 3 of 3)

- [Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments](https://mlsanity.com/paper/2502.16611/) (2025)
- [MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues](https://mlsanity.com/paper/2412.08247/) (2024)
- [Automatic Contextual Audio Denoising](https://mlsanity.com/paper/2605.22262/) (2026)

## Abstract (arXiv)

Target speaker extraction (TSE) relies on a reference cue of the target to extract the target speech from a speech mixture. While a speaker embedding is commonly used as the reference cue, such embedding pre-trained with a large number of speakers may suffer from confusion of speaker identity. In this work, we propose a multi-level speaker representation approach, from raw features to neural embeddings, to serve as the speaker reference cue. We generate a spectral-level representation from the enrollment magnitude spectrogram as a raw, low-level feature, which significantly improves the model's generalization capability. Additionally, we propose a contextual embedding feature based on cross-attention mechanisms that integrate frame-level embeddings from a pre-trained speaker encoder. By incorporating speaker features across multiple levels, we significantly enhance the performance of the TSE model. Our approach achieves a 2.74 dB improvement and a 4.94% increase in extraction accuracy on Libri2mix test set over the baseline.