# Improving Character Error Rate Is Not Equal to Having Clean Speech: Speech Enhancement for ASR Systems with Black-box Acoustic Models

arXiv: https://arxiv.org/abs/2110.05968 · HTML: https://mlsanity.com/paper/2110.05968/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [Wav2Letter: an End-to-End ConvNet-based Speech Recognition System](https://mlsanity.com/paper/1609.03193/) (2016)
- [Weighted Speech Distortion Losses for Neural-network-based Real-time Speech Enhancement](https://mlsanity.com/paper/2001.10601/) (2020)
- [Building state-of-the-art distant speech recognition using the CHiME-4 challenge with a setup of speech enhancement baseline](https://mlsanity.com/paper/1803.10109/) (2018)
- [Convolutional-Recurrent Neural Networks for Speech Enhancement](https://mlsanity.com/paper/1805.00579/) (2018)
- [Real-time Denoising and Dereverberation with Tiny Recurrent U-Net](https://mlsanity.com/paper/2102.03207/) (2021)
- [Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression](https://mlsanity.com/paper/2005.07551/) (2020)
- [Reinforcement Learning Based Speech Enhancement for Robust Speech Recognition](https://mlsanity.com/paper/1811.04224/) (2018)
- [Stable Training of DNN for Speech Enhancement based on Perceptually-Motivated Black-Box Cost Function](https://mlsanity.com/paper/2002.05879/) (2020)

## Built on by (most cited first, 2 of 2)

- [MATra: A Multilingual Attentive Transliteration System for Indian Scripts](https://mlsanity.com/paper/2208.10801/) (2022)
- [Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains](https://mlsanity.com/paper/2510.07877/) (2025)

## Abstract (arXiv)

A deep neural network (DNN)-based speech enhancement (SE) aiming to maximize the performance of an automatic speech recognition (ASR) system is proposed in this paper. In order to optimize the DNN-based SE model in terms of the character error rate (CER), which is one of the metric to evaluate the ASR system and generally non-differentiable, our method uses two DNNs: one for speech processing and one for mimicking the output CERs derived through an acoustic model (AM). Then both of DNNs are alternately optimized in the training phase. Even if the AM is a black-box, e.g., like one provided by a third-party, the proposed method enables the DNN-based SE model to be optimized in terms of the CER since the DNN mimicking the AM is differentiable. Consequently, it becomes feasible to build CER-centric SE model that has no negative effect, e.g., additional calculation cost and changing network architecture, on the inference phase since our method is merely a training scheme for the existing DNN-based methods. Experimental results show that our method improved CER by 8.8% relative derived through a black-box AM although certain noise levels are kept.