# Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancement

arXiv: https://arxiv.org/abs/2408.06911 · HTML: https://mlsanity.com/paper/2408.06911/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Omni-Dimensional Dynamic Convolution](https://mlsanity.com/paper/2209.07947/) (2022)
- [Investigating self-supervised learning for speech enhancement and separation](https://mlsanity.com/paper/2203.07960/) (2022)
- [T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement](https://mlsanity.com/paper/1910.06762/) (2019)
- [MANNER: Multi-view Attention Network for Noise Erasure](https://mlsanity.com/paper/2203.02181/) (2022)
- [DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement](https://mlsanity.com/paper/2305.08227/) (2023)
- [Boosting Self-Supervised Embeddings for Speech Enhancement](https://mlsanity.com/paper/2204.03339/) (2022)
- [CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement](https://mlsanity.com/paper/2209.11112/) (2022)
- [Dense CNN with Self-Attention for Time-Domain Speech Enhancement](https://mlsanity.com/paper/2009.01941/) (2020)
- [Self-Supervised learning with cross-modal transformers for emotion recognition](https://mlsanity.com/paper/2011.10652/) (2020)
- [Exploring WavLM on Speech Enhancement](https://mlsanity.com/paper/2211.09988/) (2022)
- [MFHCA: Enhancing Speech Emotion Recognition Via Multi-Spatial Fusion and Hierarchical Cooperative Attention](https://mlsanity.com/paper/2404.13509/) (2024)
- [Rethinking Gradient Operator for Exposing AI-enabled Face Forgeries](https://mlsanity.com/paper/2205.00767/) (2022)
- [A Recursive Network with Dynamic Attention for Monaural Speech Enhancement](https://mlsanity.com/paper/2003.12973/) (2020)

## Built on by (most cited first, 1 of 1)

- [Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study](https://mlsanity.com/paper/2503.22713/) (2025)

## Abstract (arXiv)

Self-supervised learning has demonstrated impressive performance in speech tasks, yet there remains ample opportunity for advancement in the realm of speech enhancement research. In addressing speech tasks, confining the attention mechanism solely to the temporal dimension poses limitations in effectively focusing on critical speech features. Considering the aforementioned issues, our study introduces a novel speech enhancement framework, HFSDA, which skillfully integrates heterogeneous spatial features and incorporates a dual-dimension attention mechanism to significantly enhance speech clarity and quality in noisy environments. By leveraging self-supervised learning embeddings in tandem with Short-Time Fourier Transform (STFT) spectrogram features, our model excels at capturing both high-level semantic information and detailed spectral data, enabling a more thorough analysis and refinement of speech signals. Furthermore, we employ the innovative Omni-dimensional Dynamic Convolution (ODConv) technology within the spectrogram input branch, enabling enhanced extraction and integration of crucial information across multiple dimensions. Additionally, we refine the Conformer model by enhancing its feature extraction capabilities not only in the temporal dimension but also across the spectral domain. Extensive experiments on the VCTK-DEMAND dataset show that HFSDA is comparable to existing state-of-the-art models, confirming the validity of our approach.