# Cleanformer: A multichannel array configuration-invariant neural enhancement frontend for ASR in smart speakers

arXiv: https://arxiv.org/abs/2204.11933 · HTML: https://mlsanity.com/paper/2204.11933/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 19 of 19)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Continuous Speech Separation with Conformer](https://mlsanity.com/paper/2008.05773/) (2020)
- [Toward domain-invariant speech recognition via large scale training](https://mlsanity.com/paper/1808.05312/) (2018)
- [Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments](https://mlsanity.com/paper/1705.10874/) (2017)
- [STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency](https://mlsanity.com/paper/2204.09911/) (2022)
- [Hybrid Neural Networks for On-device Directional Hearing](https://mlsanity.com/paper/2112.05893/) (2021)
- [A Neural Acoustic Echo Canceller Optimized Using An Automatic Speech Recognizer And Large Scale Synthetic Data](https://mlsanity.com/paper/2106.00856/) (2021)
- [A Neural Beam Filter for Real-time Multi-channel Speech Enhancement](https://mlsanity.com/paper/2202.02500/) (2022)

## Built on by (most cited first, 1 of 1)

- [SNR-Progressive Model with Harmonic Compensation for Low-SNR Speech Enhancement](https://mlsanity.com/paper/2406.16317/) (2024)

## Abstract (arXiv)

This work introduces the Cleanformer, a streaming multichannel neural based enhancement frontend for automatic speech recognition (ASR). This model has a conformer-based architecture which takes as inputs a single channel each of raw and enhanced signals, and uses self-attention to derive a time-frequency mask. The enhanced input is generated by a multichannel adaptive noise cancellation algorithm known as Speech Cleaner, which makes use of noise context to derive its filter taps. The time-frequency mask is applied to the noisy input to produce enhanced output features for ASR. Detailed evaluations are presented with simulated and re-recorded datasets in speech-based and non-speech-based noise that show significant reduction in word error rate (WER) when using a large-scale state-of-the-art ASR model. It also will be shown to significantly outperform enhancement using a beamformer with ideal steering. The enhancement model is agnostic of the number of microphones and array configuration and, therefore, can be used with different microphone arrays without the need for retraining. It is demonstrated that performance improves with more microphones, up to 4, with each additional microphone providing a smaller marginal benefit. Specifically, for an SNR of -6dB, relative WER improvements of about 80% are shown in both noise conditions.