# STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency

arXiv: https://arxiv.org/abs/2204.09911 · HTML: https://mlsanity.com/paper/2204.09911/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2022.

## Built on by (most cited first, 15 of 15)

- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023)
- [Iterative autoregression: a novel trick to improve your low-latency speech enhancement model](https://mlsanity.com/paper/2211.01751/) (2022)
- [DPSNN: Spiking Neural Network for Low-Latency Streaming Speech Enhancement](https://mlsanity.com/paper/2408.07388/) (2024)
- [Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement](https://mlsanity.com/paper/2411.02019/) (2024)
- [Real-Time Streamable Generative Speech Restoration with Flow Matching](https://mlsanity.com/paper/2512.19442/) (2025)
- [Real-time Low-latency Music Source Separation using Hybrid Spectrogram-TasNet](https://mlsanity.com/paper/2402.17701/) (2024)
- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)
- [Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables](https://mlsanity.com/paper/2409.18239/) (2024)
- [Speech Boosting: Low-Latency Live Speech Enhancement for TWS Earbuds](https://mlsanity.com/paper/2409.18705/) (2024)
- [TF-MLPNet: Tiny Real-Time Neural Speech Separation](https://mlsanity.com/paper/2508.03047/) (2025)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)
- [CATSE: A Context-Aware Framework for Causal Target Sound Extraction](https://mlsanity.com/paper/2403.14246/) (2024)
- [Comparison of Knowledge Distillation Methods for Low-complexity Multi-microphone Speech Enhancement using the FT-JNF Architecture](https://mlsanity.com/paper/2507.19208/) (2025)
- [Fine-grained Soundscape Control for Augmented Hearing](https://mlsanity.com/paper/2603.00395/) (2026)
- [Cloud-Boosted Low-Compute Multi-Channel Speech Enhancement](https://mlsanity.com/paper/2608.07423/) (2026)

## Abstract (arXiv)

Deep learning based speech enhancement in the short-time Fourier transform (STFT) domain typically uses a large window length such as 32 ms. A larger window can lead to higher frequency resolution and potentially better enhancement. This however incurs an algorithmic latency of 32 ms in an online setup, because the overlap-add algorithm used in the inverse STFT (iSTFT) is also performed using the same window size. To reduce this inherent latency, we adapt a conventional dual-window-size approach, where a regular input window size is used for STFT but a shorter output window is used for overlap-add, for STFT-domain deep learning based frame-online speech enhancement. Based on this STFT-iSTFT configuration, we employ complex spectral mapping for frame-online enhancement, where a deep neural network (DNN) is trained to predict the real and imaginary (RI) components of target speech from the mixture RI components. In addition, we use the DNN-predicted RI components to conduct frame-online beamforming, the results of which are used as extra features for a second DNN to perform frame-online post-filtering. The frequency-domain beamformer can be easily integrated with our DNNs and is designed to not incur any algorithmic latency. Additionally, we propose a future-frame prediction technique to further reduce the algorithmic latency. Evaluation on noisy-reverberant speech enhancement shows the effectiveness of the proposed algorithms. Compared with Conv-TasNet, our STFT-domain system can achieve better enhancement performance for a comparable amount of computation, or comparable performance with less computation, maintaining strong performance at an algorithmic latency as low as 2 ms.