# A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models

arXiv: https://arxiv.org/abs/2306.00331 · HTML: https://mlsanity.com/paper/2306.00331/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 21 of 21)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.07281/) (2020)
- [Cold Diffusion for Speech Enhancement](https://mlsanity.com/paper/2211.02527/) (2022)
- [Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder](https://mlsanity.com/paper/2102.08706/) (2021)
- [Improved Speech Enhancement with the Wave-U-Net](https://mlsanity.com/paper/1811.11307/) (2018)
- [S-DCCRN: Super Wide Band DCCRN with learnable complex feature for speech enhancement](https://mlsanity.com/paper/2111.08387/) (2021)
- [Structured State Space Decoder for Speech Recognition and Synthesis](https://mlsanity.com/paper/2210.17098/) (2022)
- [Deep neural networks for single channel source separation](https://mlsanity.com/paper/1311.2746/) (2013)

## Built on by (most cited first, 9 of 9)

- [An Investigation of Incorporating Mamba for Speech Enhancement](https://mlsanity.com/paper/2405.06573/) (2024)
- [Spiking Structured State Space Model for Monaural Speech Enhancement](https://mlsanity.com/paper/2309.03641/) (2023)
- [Exploring the Capability of Mamba in Speech Applications](https://mlsanity.com/paper/2406.16808/) (2024)
- [Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement](https://mlsanity.com/paper/2412.16626/) (2024)
- [Universal Speech Enhancement with Regression and Generative Mamba](https://mlsanity.com/paper/2505.21198/) (2025)
- [Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition](https://mlsanity.com/paper/2411.06968/) (2024)
- [MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement](https://mlsanity.com/paper/2507.00966/) (2025)
- [aTENNuate: Optimized Real-time Speech Enhancement with Deep SSMs on Raw Audio](https://mlsanity.com/paper/2409.03377/) (2024)
- [DBHN-Net: Dual-Branch Hybrid Neural Network For Low-Complexity Monaural Speech Enhancement](https://mlsanity.com/paper/2606.05911/) (2026)

## Abstract (arXiv)

We propose a multi-dimensional structured state space (S4) approach to speech enhancement. To better capture the spectral dependencies across the frequency axis, we focus on modifying the multi-dimensional S4 layer with whitening transformation to build new small-footprint models that also achieve good performance. We explore several S4-based deep architectures in time (T) and time-frequency (TF) domains. The 2-D S4 layer can be considered a particular convolutional layer with an infinite receptive field although it utilizes fewer parameters than a conventional convolutional layer. Evaluated on the VoiceBank-DEMAND data set, when compared with the conventional U-net model based on convolutional layers, the proposed TF-domain S4-based model is 78.6% smaller in size, yet it still achieves competitive results with a PESQ score of 3.15 with data augmentation. By increasing the model size, we can even reach a PESQ score of 3.18.