# T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement

arXiv: https://arxiv.org/abs/1910.06762 · HTML: https://mlsanity.com/paper/1910.06762/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2019.

## Built on by (most cited first, 30 of 34)

- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [TSTNN: Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domain](https://mlsanity.com/paper/2103.09963/) (2021)
- [Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features](https://mlsanity.com/paper/2111.02363/) (2021)
- [GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2310.05195/) (2023)
- [Transformers with Competitive Ensembles of Independent Mechanisms](https://mlsanity.com/paper/2103.00336/) (2021)
- [Dual-branch Attention-In-Attention Transformer for single-channel speech enhancement](https://mlsanity.com/paper/2110.06467/) (2021)
- [SpeechNet: A Universal Modularized Model for Speech Processing Tasks](https://mlsanity.com/paper/2105.03070/) (2021)
- [Efficient Transformer-based Speech Enhancement Using Long Frames and STFT Magnitudes](https://mlsanity.com/paper/2206.11703/) (2022)
- [OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.14183/) (2024)
- [Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks](https://mlsanity.com/paper/2101.05014/) (2021)
- [CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Framework](https://mlsanity.com/paper/2503.02593/) (2025)
- [InQSS: a speech intelligibility and quality assessment model using a multi-task learning network](https://mlsanity.com/paper/2111.02585/) (2021)
- [ECG Artifact Removal from Single-Channel Surface EMG Using Fully Convolutional Networks](https://mlsanity.com/paper/2210.13271/) (2022)
- [Perceptual Contrast Stretching on Target Feature for Speech Enhancement](https://mlsanity.com/paper/2203.17152/) (2022)
- [MTI-Net: A Multi-Target Speech Intelligibility Prediction Model](https://mlsanity.com/paper/2204.03310/) (2022)
- [Exploring Self-Attention Mechanisms for Speech Separation](https://mlsanity.com/paper/2202.02884/) (2022)
- [Deep Sound Field Reconstruction in Real Rooms: Introducing the ISOBEL Sound Field Dataset](https://mlsanity.com/paper/2102.06455/) (2021)
- [Single-channel speech enhancement using learnable loss mixup](https://mlsanity.com/paper/2312.17255/) (2023)
- [Density Adaptive Attention is All You Need: Robust Parameter-Efficient Fine-Tuning Across Multiple Modalities](https://mlsanity.com/paper/2401.11143/) (2024)
- [Multi-Task Pseudo-Label Learning for Non-Intrusive Speech Quality Assessment Model](https://mlsanity.com/paper/2308.09262/) (2023)
- [Toward a Deeper Understanding: RetNet Viewed through Convolution](https://mlsanity.com/paper/2309.05375/) (2023)
- [Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancement](https://mlsanity.com/paper/2408.06911/) (2024)
- [Selective State Space Model for Monaural Speech Enhancement](https://mlsanity.com/paper/2411.06217/) (2024)
- [Speech Enhancement using Separable Polling Attention and Global Layer Normalization followed with PReLU](https://mlsanity.com/paper/2105.02509/) (2021)
- [Full Attention Bidirectional Deep Learning Structure for Single Channel Speech Enhancement](https://mlsanity.com/paper/2108.12105/) (2021)

## Abstract (arXiv)

Transformer neural networks (TNN) demonstrated state-of-art performance on many natural language processing (NLP) tasks, replacing recurrent neural networks (RNNs), such as LSTMs or GRUs. However, TNNs did not perform well in speech enhancement, whose contextual nature is different than NLP tasks, like machine translation. Self-attention is a core building block of the Transformer, which not only enables parallelization of sequence computation, but also provides the constant path length between symbols that is essential to learning long-range dependencies. In this paper, we propose a Transformer with Gaussian-weighted self-attention (T-GSA), whose attention weights are attenuated according to the distance between target and context symbols. The experimental results show that the proposed T-GSA has significantly improved speech-enhancement performance, compared to the Transformer and RNNs.