# Metric-oriented Speech Enhancement using Diffusion Probabilistic Model

arXiv: https://arxiv.org/abs/2302.11989 · HTML: https://mlsanity.com/paper/2302.11989/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [A Study of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1808.08866/) (2018)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data](https://mlsanity.com/paper/2203.15321/) (2022)
- [Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention](https://mlsanity.com/paper/2002.05873/) (2020)
- [Improving GANs for Speech Enhancement](https://mlsanity.com/paper/2001.05532/) (2020)
- [Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning](https://mlsanity.com/paper/2210.15324/) (2022)
- [Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning](https://mlsanity.com/paper/2212.05301/) (2022)
- [Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2110.05267/) (2021)
- [A Flow-Based Neural Network for Time Domain Speech Enhancement](https://mlsanity.com/paper/2106.09008/) (2021)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021](https://mlsanity.com/paper/2107.00279/) (2021)
- [WaveCRN: An Efficient Convolutional Recurrent Neural Network for End-to-end Speech Enhancement](https://mlsanity.com/paper/2004.04098/) (2020)
- [Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2203.14838/) (2022)
- [Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separation](https://mlsanity.com/paper/2302.11131/) (2023)
- [Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2302.11362/) (2023)
- [Sequence-to-Sequence ASR Optimization via Reinforcement Learning](https://mlsanity.com/paper/1710.10774/) (2017)
- [Self-critical Sequence Training for Automatic Speech Recognition](https://mlsanity.com/paper/2204.06260/) (2022)

## Built on by (most cited first, 11 of 11)

- [GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators](https://mlsanity.com/paper/2402.06894/) (2024)
- [Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR](https://mlsanity.com/paper/2304.04974/) (2023)
- [Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders](https://mlsanity.com/paper/2305.10734/) (2023)
- [Investigating the Design Space of Diffusion Models for Speech Enhancement](https://mlsanity.com/paper/2312.04370/) (2023)
- [Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2305.09212/) (2023)
- [Noise-aware Speech Enhancement using Diffusion Probabilistic Model](https://mlsanity.com/paper/2307.08029/) (2023)
- [Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions Using a Heun-Based Sampler](https://mlsanity.com/paper/2312.02683/) (2023)
- [Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10563/) (2023)
- [A Variance-Preserving Interpolation Approach for Diffusion Models with Applications to Single Channel Speech Enhancement and Recognition](https://mlsanity.com/paper/2405.16952/) (2024)
- [MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10567/) (2023)
- [The Effect of Training Dataset Size on Discriminative and Diffusion-Based Speech Enhancement Systems](https://mlsanity.com/paper/2406.06160/) (2024)

## Abstract (arXiv)

Deep neural network based speech enhancement technique focuses on learning a noisy-to-clean transformation supervised by paired training data. However, the task-specific evaluation metric (e.g., PESQ) is usually non-differentiable and can not be directly constructed in the training criteria. This mismatch between the training objective and evaluation metric likely results in sub-optimal performance. To alleviate it, we propose a metric-oriented speech enhancement method (MOSE), which leverages the recent advances in the diffusion probabilistic model and integrates a metric-oriented training strategy into its reverse process. Specifically, we design an actor-critic based framework that considers the evaluation metric as a posterior reward, thus guiding the reverse process to the metric-increasing direction. The experimental results demonstrate that MOSE obviously benefits from metric-oriented training and surpasses the generative baselines in terms of all evaluation metrics.