# Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention

arXiv: https://arxiv.org/abs/2002.05873 · HTML: https://mlsanity.com/paper/2002.05873/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2020.

## Built on by (most cited first, 28 of 28)

- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [An Investigation of Incorporating Mamba for Speech Enhancement](https://mlsanity.com/paper/2405.06573/) (2024)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [Hybrid Relation Guided Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2204.13423/) (2022)
- [Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features](https://mlsanity.com/paper/2111.02363/) (2021)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Dual-branch Attention-In-Attention Transformer for single-channel speech enhancement](https://mlsanity.com/paper/2110.06467/) (2021)
- [Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport](https://mlsanity.com/paper/2111.06316/) (2021)
- [SepTr: Separable Transformer for Audio Spectrogram Processing](https://mlsanity.com/paper/2203.09581/) (2022)
- [HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2301.03330/) (2023)
- [Metric-oriented Speech Enhancement using Diffusion Probabilistic Model](https://mlsanity.com/paper/2302.11989/) (2023)
- [Self-Attention for Audio Super-Resolution](https://mlsanity.com/paper/2108.11637/) (2021)
- [Monaural Speech Enhancement with Complex Convolutional Block Attention Module and Joint Time Frequency Losses](https://mlsanity.com/paper/2102.01993/) (2021)
- [MIMO Self-attentive RNN Beamformer for Multi-speaker Speech Separation](https://mlsanity.com/paper/2104.08450/) (2021)
- [Zero-Shot Personalized Speech Enhancement through Speaker-Informed Model Selection](https://mlsanity.com/paper/2105.03542/) (2021)
- [MUSE: Flexible Voiceprint Receptive Fields and Multi-Path Fusion Enhanced Taylor Transformer for U-Net-based Speech Enhancement](https://mlsanity.com/paper/2406.04589/) (2024)
- [WASE: Learning When to Attend for Speaker Extraction in Cocktail Party Environments](https://mlsanity.com/paper/2106.07016/) (2021)
- [Tune-In: Training Under Negative Environments with Interference for Attention Networks Simulating Cocktail Party Effect](https://mlsanity.com/paper/2103.01461/) (2021)
- [Time-domain Speech Enhancement with Generative Adversarial Learning](https://mlsanity.com/paper/2103.16149/) (2021)
- [Attention-Based Acoustic Feature Fusion Network for Depression Detection](https://mlsanity.com/paper/2308.12478/) (2023)
- [A Training and Inference Strategy Using Noisy and Enhanced Speech as Target for Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2210.15368/) (2022)
- [Selective State Space Model for Monaural Speech Enhancement](https://mlsanity.com/paper/2411.06217/) (2024)
- [A Monotonicity Constrained Attention Module for Emotion Classification with Limited EEG Data](https://mlsanity.com/paper/2208.08155/) (2022)
- [Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement](https://mlsanity.com/paper/2409.09642/) (2024)
- [A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition](https://mlsanity.com/paper/2502.16762/) (2025)
- [Masked Autoencoders as Universal Speech Enhancer](https://mlsanity.com/paper/2602.02413/) (2026)
- [DBHN-Net: Dual-Branch Hybrid Neural Network For Low-Complexity Monaural Speech Enhancement](https://mlsanity.com/paper/2606.05911/) (2026)
- [RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation](https://mlsanity.com/paper/2608.12099/) (2026)

## Abstract (arXiv)

This paper investigates a self-adaptation method for speech enhancement using auxiliary speaker-aware features; we extract a speaker representation used for adaptation directly from the test utterance. Conventional studies of deep neural network (DNN)--based speech enhancement mainly focus on building a speaker independent model. Meanwhile, in speech applications including speech recognition and synthesis, it is known that model adaptation to the target speaker improves the accuracy. Our research question is whether a DNN for speech enhancement can be adopted to unknown speakers without any auxiliary guidance signal in test-phase. To achieve this, we adopt multi-task learning of speech enhancement and speaker identification, and use the output of the final hidden layer of speaker identification branch as an auxiliary feature. In addition, we use multi-head self-attention for capturing long-term dependencies in the speech and noise. Experimental results on a public dataset show that our strategy achieves the state-of-the-art performance and also outperform conventional methods in terms of subjective quality.