# End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1709.03658 · HTML: https://mlsanity.com/paper/1709.03658/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2017.

## Built on by (most cited first, 26 of 26)

- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [An Investigation of Incorporating Mamba for Speech Enhancement](https://mlsanity.com/paper/2405.06573/) (2024)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement](https://mlsanity.com/paper/2209.11112/) (2022)
- [Perceive and predict: self-supervised speech representation based loss functions for speech enhancement](https://mlsanity.com/paper/2301.04388/) (2023)
- [Speech Enhancement Using Continuous Embeddings of Neural Audio Codec](https://mlsanity.com/paper/2502.16240/) (2025)
- [Perceptual Contrast Stretching on Target Feature for Speech Enhancement](https://mlsanity.com/paper/2203.17152/) (2022)
- [End-to-End Learning for Stochastic Optimization: A Bayesian Perspective](https://mlsanity.com/paper/2306.04174/) (2023)
- [FlowTSE: Target Speaker Extraction with Flow Matching](https://mlsanity.com/paper/2505.14465/) (2025)
- [MetricGAN+/-: Increasing Robustness of Noise Reduction on Unseen Data](https://mlsanity.com/paper/2203.12369/) (2022)
- [Variance-Preserving-Based Interpolation Diffusion Models for Speech Enhancement](https://mlsanity.com/paper/2306.08527/) (2023)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Blind Restoration of Real-World Audio by 1D Operational GANs](https://mlsanity.com/paper/2212.14618/) (2022)
- [Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss](https://mlsanity.com/paper/2103.01463/) (2021)
- [Multi-Loss Convolutional Network with Time-Frequency Attention for Speech Enhancement](https://mlsanity.com/paper/2306.08956/) (2023)
- [Towards Intelligibility-Oriented Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2111.09642/) (2021)
- [A Novel Speech Intelligibility Enhancement Model based on CanonicalCorrelation and Deep Learning](https://mlsanity.com/paper/2202.05756/) (2022)
- [D4AM: A General Denoising Framework for Downstream Acoustic Models](https://mlsanity.com/paper/2311.16595/) (2023)
- [Selective State Space Model for Monaural Speech Enhancement](https://mlsanity.com/paper/2411.06217/) (2024)
- [DeepExtractor: Time-domain reconstruction of signals and glitches in gravitational wave data with deep learning](https://mlsanity.com/paper/2501.18423/) (2025)
- [SEOFP-NET: Compression and Acceleration of Deep Neural Networks for Speech Enhancement Using Sign-Exponent-Only Floating-Points](https://mlsanity.com/paper/2111.04436/) (2021)
- [Capacity-Net-Based RIS Precoding Design without Channel Estimation for mmWave MIMO System](https://mlsanity.com/paper/2509.25660/) (2025)
- [Multi-channel multi-speaker transformer for speech recognition](https://mlsanity.com/paper/2601.02688/) (2026)
- [RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation](https://mlsanity.com/paper/2608.12099/) (2026)

## Abstract (arXiv)

Speech enhancement model is used to map a noisy speech to a clean speech. In the training stage, an objective function is often adopted to optimize the model parameters. However, in most studies, there is an inconsistency between the model optimization criterion and the evaluation criterion on the enhanced speech. For example, in measuring speech intelligibility, most of the evaluation metric is based on a short-time objective intelligibility (STOI) measure, while the frame based minimum mean square error (MMSE) between estimated and clean speech is widely used in optimizing the model. Due to the inconsistency, there is no guarantee that the trained model can provide optimal performance in applications. In this study, we propose an end-to-end utterance-based speech enhancement framework using fully convolutional neural networks (FCN) to reduce the gap between the model optimization and evaluation criterion. Because of the utterance-based optimization, temporal correlation information of long speech segments, or even at the entire utterance level, can be considered when perception-based objective functions are used for the direct optimization. As an example, we implement the proposed FCN enhancement framework to optimize the STOI measure. Experimental results show that the STOI of test speech is better than conventional MMSE-optimized speech due to the consistency between the training and evaluation target. Moreover, by integrating the STOI in model optimization, the intelligibility of human subjects and automatic speech recognition (ASR) system on the enhanced speech is also substantially improved compared to those generated by the MMSE criterion.