# A Framework for Unified Real-time Personalized and Non-Personalized Speech Enhancement

arXiv: https://arxiv.org/abs/2302.11768 · HTML: https://mlsanity.com/paper/2302.11768/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2023.

## Built on by (most cited first, 2 of 2)

- [Personalized Speech Enhancement Without a Separate Speaker Embedding Model](https://mlsanity.com/paper/2406.09928/) (2024)
- [Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement](https://mlsanity.com/paper/2409.09642/) (2024)

## Abstract (arXiv)

In this study, we present an approach to train a single speech enhancement network that can perform both personalized and non-personalized speech enhancement. This is achieved by incorporating a frame-wise conditioning input that specifies the type of enhancement output. To improve the quality of the enhanced output and mitigate oversuppression, we experiment with re-weighting frames by the presence or absence of speech activity and applying augmentations to speaker embeddings. By training under a multi-task learning setting, we empirically show that the proposed unified model obtains promising results on both personalized and non-personalized speech enhancement benchmarks and reaches similar performance to models that are trained specialized for either task. The strong performance of the proposed method demonstrates that the unified model is a more economical alternative compared to keeping separate task-specific models during inference.