# Unsupervised Speech Enhancement with speech recognition embedding and disentanglement losses

arXiv: https://arxiv.org/abs/2111.08678 · HTML: https://mlsanity.com/paper/2111.08678/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2021.

## Built on by (most cited first, 7 of 7)

- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders](https://mlsanity.com/paper/2404.02702/) (2024)
- [Gaussian Flow Bridges for Audio Domain Transfer with Unpaired Data](https://mlsanity.com/paper/2405.19497/) (2024)
- [A Training and Inference Strategy Using Noisy and Enhanced Speech as Target for Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2210.15368/) (2022)
- [DSNet: Disentangled Siamese Network with Neutral Calibration for Speech Emotion Recognition](https://mlsanity.com/paper/2312.15593/) (2023)
- [Speech Separation based on Contrastive Learning and Deep Modularization](https://mlsanity.com/paper/2305.10652/) (2023)

## Abstract (arXiv)

Speech enhancement has recently achieved great success with various deep learning methods. However, most conventional speech enhancement systems are trained with supervised methods that impose two significant challenges. First, a majority of training datasets for speech enhancement systems are synthetic. When mixing clean speech and noisy corpora to create the synthetic datasets, domain mismatches occur between synthetic and real-world recordings of noisy speech or audio. Second, there is a trade-off between increasing speech enhancement performance and degrading speech recognition (ASR) performance. Thus, we propose an unsupervised loss function to tackle those two problems. Our function is developed by extending the MixIT loss function with speech recognition embedding and disentanglement loss. Our results show that the proposed function effectively improves the speech enhancement performance compared to a baseline trained in a supervised way on the noisy VoxCeleb dataset. While fully unsupervised training is unable to exceed the corresponding baseline, with joint super- and unsupervised training, the system is able to achieve similar speech quality and better ASR performance than the best supervised baseline.