# Decoupled Spatial and Temporal Processing for Resource Efficient Multichannel Speech Enhancement

arXiv: https://arxiv.org/abs/2401.07879 · HTML: https://mlsanity.com/paper/2401.07879/

## Key facts

- Cited by 2 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation](https://mlsanity.com/paper/2209.03952/) (2022)
- [End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation](https://mlsanity.com/paper/1910.14104/) (2019)
- [Channel-Attention Dense U-Net for Multichannel Speech Enhancement](https://mlsanity.com/paper/2001.11542/) (2020)
- [Dense CNN with Self-Attention for Time-Domain Speech Enhancement](https://mlsanity.com/paper/2009.01941/) (2020)
- [Raw Waveform-based Speech Enhancement by Fully Convolutional Networks](https://mlsanity.com/paper/1703.02205/) (2017)
- [STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency](https://mlsanity.com/paper/2204.09911/) (2022)
- [Complex spectrogram enhancement by convolutional neural network with multi-metrics learning](https://mlsanity.com/paper/1704.08504/) (2017)
- [TPARN: Triple-path Attentive Recurrent Network for Time-domain Multichannel Speech Enhancement](https://mlsanity.com/paper/2110.10757/) (2021)
- [On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environments](https://mlsanity.com/paper/2011.05958/) (2020)
- [Multichannel Speech Enhancement by Raw Waveform-mapping using Fully Convolutional Networks](https://mlsanity.com/paper/1909.11909/) (2019)
- [DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement](https://mlsanity.com/paper/2212.07570/) (2022)
- [Multichannel Speech Enhancement without Beamforming](https://mlsanity.com/paper/2110.13130/) (2021)
- [Time-domain Ad-hoc Array Speech Enhancement Using a Triple-path Network](https://mlsanity.com/paper/2110.11844/) (2021)
- [Neural Speech Enhancement with Very Low Algorithmic Latency and Complexity via Integrated Full- and Sub-Band Modeling](https://mlsanity.com/paper/2304.08707/) (2023)
- [UX-NET: Filter-and-Process-based Improved U-Net for Real-time Time-domain Audio Separation](https://mlsanity.com/paper/2210.15822/) (2022)

## Built on by (most cited first, 2 of 2)

- [Improving Resource-Efficient Speech Enhancement via Neural Differentiable DSP Vocoder Refinement](https://mlsanity.com/paper/2508.14709/) (2025)
- [Direction-Preserving MIMO Speech Enhancement Using a Neural Covariance Estimator](https://mlsanity.com/paper/2604.11179/) (2026)

## Abstract (arXiv)

We present a novel model designed for resource-efficient multichannel speech enhancement in the time domain, with a focus on low latency, lightweight, and low computational requirements. The proposed model incorporates explicit spatial and temporal processing within deep neural network (DNN) layers. Inspired by frequency-dependent multichannel filtering, our spatial filtering process applies multiple trainable filters to each hidden unit across the spatial dimension, resulting in a multichannel output. The temporal processing is applied over a single-channel output stream from the spatial processing using a Long Short-Term Memory (LSTM) network. The output from the temporal processing stage is then further integrated into the spatial dimension through elementwise multiplication. This explicit separation of spatial and temporal processing results in a resource-efficient network design. Empirical findings from our experiments show that our proposed model significantly outperforms robust baseline models while demanding far fewer parameters and computations, while achieving an ultra-low algorithmic latency of just 2 milliseconds.