# Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement

arXiv: https://arxiv.org/abs/2411.02019 · HTML: https://mlsanity.com/paper/2411.02019/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement](https://mlsanity.com/paper/1709.08243/) (2017)
- [Spartus: A 9.4 TOp/s FPGA-based LSTM Accelerator Exploiting Spatio-Temporal Sparsity](https://mlsanity.com/paper/2108.02297/) (2021)
- [Delta Networks for Optimized Recurrent Network Computation](https://mlsanity.com/paper/1612.05571/) (2016)
- [Dense CNN with Self-Attention for Time-Domain Speech Enhancement](https://mlsanity.com/paper/2009.01941/) (2020)
- [STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency](https://mlsanity.com/paper/2204.09911/) (2022)
- [Ultra Low Complexity Deep Learning Based Noise Suppression](https://mlsanity.com/paper/2312.08132/) (2023)
- [Deep neural network Based Low-latency Speech Separation with Asymmetric analysis-Synthesis Window Pair](https://mlsanity.com/paper/2106.11794/) (2021)
- [V-SlowFast Network for Efficient Visual Sound Separation](https://mlsanity.com/paper/2109.08867/) (2021)
- [Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancement](https://mlsanity.com/paper/2408.12425/) (2024)
- [Knowledge boosting during low-latency inference](https://mlsanity.com/paper/2407.11055/) (2024)
- [Unsupervised Low Latency Speech Enhancement with RT-GCC-NMF](https://mlsanity.com/paper/1904.03130/) (2019)
- [Real-time speech enhancement using equilibriated RNN](https://mlsanity.com/paper/2002.05843/) (2020)

## Built on by (most cited first, 4 of 4)

- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)
- [Towards Lightweight Adaptation of Speech Enhancement Models in Real-World Environments](https://mlsanity.com/paper/2603.07471/) (2026)
- [Cloud-Boosted Low-Compute Multi-Channel Speech Enhancement](https://mlsanity.com/paper/2608.07423/) (2026)
- [Transformer-based Neural Beamforming for Real-Time Speech Enhancement on Smart Low-Power Hearable Devices](https://mlsanity.com/paper/2609.33755/) (2026)

## Abstract (arXiv)

Deep learning-based speech enhancement (SE) methods often face significant computational challenges when needing to meet low-latency requirements because of the increased number of frames to be processed. This paper introduces the SlowFast framework which aims to reduce computation costs specifically when low-latency enhancement is needed. The framework consists of a slow branch that analyzes the acoustic environment at a low frame rate, and a fast branch that performs SE in the time domain at the needed higher frame rate to match the required latency. Specifically, the fast branch employs a state space model where its state transition process is dynamically modulated by the slow branch. Experiments on a SE task with a 2 ms algorithmic latency requirement using the Voice Bank + Demand dataset show that our approach reduces computation cost by 70% compared to a baseline single-branch network with equivalent parameters, without compromising enhancement performance. Furthermore, by leveraging the SlowFast framework, we implemented a network that achieves an algorithmic latency of just 62.5 μs (one sample point at 16 kHz sample rate) with a computation cost of 100 M MACs/s, while scoring a PESQ-NB of 3.12 and SISNR of 16.62.