# SEF-PNet: Speaker Encoder-Free Personalized Speech Enhancement with Local and Global Contexts Aggregation

arXiv: https://arxiv.org/abs/2501.11274 · HTML: https://mlsanity.com/paper/2501.11274/

## Key facts

- Cited by 5 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [ICASSP 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2009.06122/) (2020)
- [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [High Fidelity Speech Enhancement with Band-split RNN](https://mlsanity.com/paper/2212.00406/) (2022)
- [X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion](https://mlsanity.com/paper/2303.05023/) (2023)
- [TEA-PSE 3.0: Tencent-Ethereal-Audio-Lab Personalized Speech Enhancement System For ICASSP 2023 DNS Challenge](https://mlsanity.com/paper/2303.07704/) (2023)
- [DPCCN: Densely-Connected Pyramid Complex Convolutional Network for Robust Speech Separation And Extraction](https://mlsanity.com/paper/2112.13520/) (2021)
- [Target Speaker Extraction by Directly Exploiting Contextual Information in the Time-Frequency Domain](https://mlsanity.com/paper/2402.17146/) (2024)
- [Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches](https://mlsanity.com/paper/2204.01355/) (2022)
- [Hierarchical speaker representation for target speaker extraction](https://mlsanity.com/paper/2210.15849/) (2022)
- [MC-SpEx: Towards Effective Speaker Extraction with Multi-Scale Interfusion and Conditional Speaker Modulation](https://mlsanity.com/paper/2306.16250/) (2023)
- [Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive Feature Learning in Speech Enhancement](https://mlsanity.com/paper/2306.05861/) (2023)
- [An Efficient Speech Separation Network Based on Recurrent Fusion Dilated Convolution and Channel Attention](https://mlsanity.com/paper/2306.05887/) (2023)

## Built on by (most cited first, 5 of 5)

- [Unified Architecture and Unsupervised Speech Disentanglement for Speaker Embedding-Free Enrollment in Personalized Speech Enhancement](https://mlsanity.com/paper/2505.12288/) (2025)
- [Lightweight speech enhancement guided target speech extraction in noisy multi-speaker scenarios](https://mlsanity.com/paper/2508.19583/) (2025)
- [TripleC Learning and Lightweight Speech Enhancement for Multi-Condition Target Speech Extraction](https://mlsanity.com/paper/2512.04945/) (2025)
- [Enroll-on-Wakeup: A First Comparative Study of Target Speech Extraction for Seamless Interaction in Real Noisy Human-Machine Dialogue Scenarios](https://mlsanity.com/paper/2602.15519/) (2026)
- [Unified Target-Speaker ASR with Text and Enrollment Speech Cues](https://mlsanity.com/paper/2609.33853/) (2026)

## Abstract (arXiv)

Personalized speech enhancement (PSE) methods typically rely on pre-trained speaker verification models or self-designed speaker encoders to extract target speaker clues, guiding the PSE model in isolating the desired speech. However, these approaches suffer from significant model complexity and often underutilize enrollment speaker information, limiting the potential performance of the PSE model. To address these limitations, we propose a novel Speaker Encoder-Free PSE network, termed SEF-PNet, which fully exploits the information present in both the enrollment speech and noisy mixtures. SEF-PNet incorporates two key innovations: Interactive Speaker Adaptation (ISA) and Local-Global Context Aggregation (LCA). ISA dynamically modulates the interactions between enrollment and noisy signals to enhance the speaker adaptation, while LCA employs advanced channel attention within the PSE encoder to effectively integrate local and global contextual information, thus improving feature learning. Experiments on the Libri2Mix dataset demonstrate that SEF-PNet significantly outperforms baseline models, achieving state-of-the-art PSE performance.