# Mask scalar prediction for improving robust automatic speech recognition

arXiv: https://arxiv.org/abs/2204.12092 · HTML: https://mlsanity.com/paper/2204.12092/

## Key facts

- Cited by 3 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 21 of 21)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR](https://mlsanity.com/paper/2201.06685/) (2022)
- [Toward domain-invariant speech recognition via large scale training](https://mlsanity.com/paper/1808.05312/) (2018)
- [Improving noise robust automatic speech recognition with single-channel time-domain enhancement network](https://mlsanity.com/paper/2003.03998/) (2020)
- [Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments](https://mlsanity.com/paper/1705.10874/) (2017)
- [Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition](https://mlsanity.com/paper/2201.03881/) (2022)
- [Dual Application of Speech Enhancement for Automatic Speech Recognition](https://mlsanity.com/paper/2011.03840/) (2020)
- [Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognition](https://mlsanity.com/paper/2106.00949/) (2021)
- [SNRi Target Training for Joint Speech Enhancement and Recognition](https://mlsanity.com/paper/2111.00764/) (2021)
- [A Neural Acoustic Echo Canceller Optimized Using An Automatic Speech Recognizer And Large Scale Synthetic Data](https://mlsanity.com/paper/2106.00856/) (2021)
- [Task-aware Warping Factors in Mask-based Speech Enhancement](https://mlsanity.com/paper/2108.12128/) (2021)

## Built on by (most cited first, 3 of 3)

- [speech and noise dual-stream spectrogram refine network with speech distortion loss for robust speech recognition](https://mlsanity.com/paper/2305.17860/) (2023)
- [A Conformer-based Waveform-domain Neural Acoustic Echo Canceller Optimized for ASR Accuracy](https://mlsanity.com/paper/2205.03481/) (2022)
- [Where Speech Enhancement Hurts Recognition: An Inference Time Polar Projection Diagnosis](https://mlsanity.com/paper/2607.11157/) (2026)

## Abstract (arXiv)

Using neural network based acoustic frontends for improving robustness of streaming automatic speech recognition (ASR) systems is challenging because of the causality constraints and the resulting distortion that the frontend processing introduces in speech. Time-frequency masking based approaches have been shown to work well, but they need additional hyper-parameters to scale the mask to limit speech distortion. Such mask scalars are typically hand-tuned and chosen conservatively. In this work, we present a technique to predict mask scalars using an ASR-based loss in an end-to-end fashion, with minimal increase in the overall model size and complexity. We evaluate the approach on two robust ASR tasks: multichannel enhancement in the presence of speech and non-speech noise, and acoustic echo cancellation (AEC). Results show that the presented algorithm consistently improves word error rate (WER) without the need for any additional tuning over strong baselines that use hand-tuned hyper-parameters: up to 16% for multichannel enhancement in noisy conditions, and up to 7% for AEC.