# DNN-Free Low-Latency Adaptive Speech Enhancement Based on Frame-Online Beamforming Powered by Block-Online FastMNMF

arXiv: https://arxiv.org/abs/2207.10934 · HTML: https://mlsanity.com/paper/2207.10934/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 7 of 7)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Unsupervised Speech Enhancement Based on Multichannel NMF-Informed Beamforming for Noise-Robust Automatic Speech Recognition](https://mlsanity.com/paper/1903.09341/) (2019)
- [NICE-Beam: Neural Integrated Covariance Estimators for Time-Varying Beamformers](https://mlsanity.com/paper/2112.04613/) (2021)
- [Complex Neural Spatial Filter: Enhancing Multi-channel Target Speech Separation in Complex Domain](https://mlsanity.com/paper/2104.12359/) (2021)
- [Customizable End-to-end Optimization of Online Neural Network-supported Dereverberation for Hearing Devices](https://mlsanity.com/paper/2204.02694/) (2022)
- [Direction-Aware Adaptive Online Neural Speech Enhancement with an Augmented Reality Headset in Real Noisy Conversational Environments](https://mlsanity.com/paper/2207.07296/) (2022)
- [SLOGD: Speaker LOcation Guided Deflation approach to speech separation](https://mlsanity.com/paper/1910.11131/) (2019)

## Abstract (arXiv)

This paper describes a practical dual-process speech enhancement system that adapts environment-sensitive frame-online beamforming (front-end) with help from environment-free block-online source separation (back-end). To use minimum variance distortionless response (MVDR) beamforming, one may train a deep neural network (DNN) that estimates time-frequency masks used for computing the covariance matrices of sources (speech and noise). Backpropagation-based run-time adaptation of the DNN was proposed for dealing with the mismatched training-test conditions. Instead, one may try to directly estimate the source covariance matrices with a state-of-the-art blind source separation method called fast multichannel non-negative matrix factorization (FastMNMF). In practice, however, neither the DNN nor the FastMNMF can be updated in a frame-online manner due to its computationally-expensive iterative nature. Our DNN-free system leverages the posteriors of the latest source spectrograms given by block-online FastMNMF to derive the current source covariance matrices for frame-online beamforming. The evaluation shows that our frame-online system can quickly respond to scene changes caused by interfering speaker movements and outperformed an existing block-online system with DNN-based beamforming by 5.0 points in terms of the word error rate.