# Improving Frame-Online Neural Speech Enhancement with Overlapped-Frame Prediction

arXiv: https://arxiv.org/abs/2204.07566 · HTML: https://mlsanity.com/paper/2204.07566/

## Key facts

- Cited by 5 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 24 of 24)

- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [Towards efficient models for real-time deep noise suppression](https://mlsanity.com/paper/2101.09249/) (2021)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)
- [Weighted Speech Distortion Losses for Neural-network-based Real-time Speech Enhancement](https://mlsanity.com/paper/2001.10601/) (2020)
- [Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation](https://mlsanity.com/paper/2010.01703/) (2020)
- [On The Compensation Between Magnitude and Phase in Speech Separation](https://mlsanity.com/paper/2108.05470/) (2021)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Framework](https://mlsanity.com/paper/2001.08662/) (2020)
- [Complex spectrogram enhancement by convolutional neural network with multi-metrics learning](https://mlsanity.com/paper/1704.08504/) (2017)
- [Real-time binaural speech separation with preserved spatial cues](https://mlsanity.com/paper/2002.06637/) (2020)
- [ICASSP 2021 Deep Noise Suppression Challenge: Decoupling Magnitude and Phase Optimization with a Two-Stage Deep Network](https://mlsanity.com/paper/2102.04198/) (2021)
- [Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation](https://mlsanity.com/paper/2108.07376/) (2021)
- [SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separation](https://mlsanity.com/paper/2201.10800/) (2022)
- [ICASSP 2022 Acoustic Echo Cancellation Challenge](https://mlsanity.com/paper/2202.13290/) (2022)
- [Exploring Tradeoffs in Models for Low-latency Speech Enhancement](https://mlsanity.com/paper/1811.07030/) (2018)
- [Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement](https://mlsanity.com/paper/2110.00570/) (2021)
- [Low-Latency Speaker-Independent Continuous Speech Separation](https://mlsanity.com/paper/1904.06478/) (2019)
- [INTERSPEECH 2021 ConferencingSpeech Challenge: Towards Far-field Multi-Channel Speech Enhancement for Video Conferencing](https://mlsanity.com/paper/2104.00960/) (2021)

## Built on by (most cited first, 5 of 5)

- [DualVC 3: Leveraging Language Model Generated Pseudo Context for End-to-end Low Latency Streaming Voice Conversion](https://mlsanity.com/paper/2406.07846/) (2024)
- [End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations](https://mlsanity.com/paper/2303.12002/) (2023)
- [Causal Signal-Based DCCRN with Overlapped-Frame Prediction for Online Speech Enhancement](https://mlsanity.com/paper/2309.03684/) (2023)
- [Speech Enhancement with Overlapped-Frame Information Fusion and Causal Self-Attention](https://mlsanity.com/paper/2501.12004/) (2025)
- [HALO: Half-Frame-Rate Adaptive Learnable Operator for Lightweight STFT-Based Speech Enhancement](https://mlsanity.com/paper/2606.12328/) (2026)

## Abstract (arXiv)

Frame-online speech enhancement systems in the short-time Fourier transform (STFT) domain usually have an algorithmic latency equal to the window size due to the use of overlap-add in the inverse STFT (iSTFT). This algorithmic latency allows the enhancement models to leverage future contextual information up to a length equal to the window size. However, this information is only partially leveraged by current frame-online systems. To fully exploit it, we propose an overlapped-frame prediction technique for deep learning based frame-online speech enhancement, where at each frame our deep neural network (DNN) predicts the current and several past frames that are necessary for overlap-add, instead of only predicting the current frame. In addition, we propose a loss function to account for the scale difference between predicted and oracle target signals. Experiments on a noisy-reverberant speech enhancement task show the effectiveness of the proposed algorithms.