mlsanityData to 14 Oct 2026

T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement

arXiv 1910.06762 ↗First posted 13 Oct 2019eess.AS cs.SDcited by 34

Not on the map. Why?

Abstract (arXiv)

Transformer neural networks (TNN) demonstrated state-of-art performance on many natural language processing (NLP) tasks, replacing recurrent neural networks (RNNs), such as LSTMs or GRUs. However, TNNs did not perform well in speech enhancement, whose contextual nature is different than NLP tasks, like machine translation. Self-attention is a core building block of the Transformer, which not only enables parallelization of sequence computation, but also provides the constant path length between symbols that is essential to learning long-range dependencies. In this paper, we propose a Transformer with Gaussian-weighted self-attention (T-GSA), whose attention weights are attenuated according to the distance between target and context symbols. The experimental results show that the proposed T-GSA has significantly improved speech-enhancement performance, compared to the Transformer and RNNs.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 7; 2023: 5; 2024: 7; 2025: 2.

34 papers.

Sort:

By area:

  1. 2021Conditional DETR for Fast Training Convergencecited by 424
  2. 2021A Survey of Transformerscited by 401
  3. 2022MetaFormer Baselines for Visioncited by 132
  4. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 60
  5. 2023Transformers in Speech Processing: A Surveycited by 39
  6. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 37
  7. 2021TSTNN: Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domaincited by 36
  8. 2021Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Featurescited by 33
  9. 2023GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrievalcited by 20
  10. 2021Transformers with Competitive Ensembles of Independent Mechanismscited by 19
  11. 2021Dual-branch Attention-In-Attention Transformer for single-channel speech enhancementcited by 18
  12. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16
22 more
  1. 2022Efficient Transformer-based Speech Enhancement Using Long Frames and STFT Magnitudescited by 10
  2. 2024OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentationcited by 9
  3. 2021Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networkscited by 8
  4. 2025CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Frameworkcited by 8
  5. 2021InQSS: a speech intelligibility and quality assessment model using a multi-task learning networkcited by 7
  6. 2022ECG Artifact Removal from Single-Channel Surface EMG Using Fully Convolutional Networkscited by 7
  7. 2022Perceptual Contrast Stretching on Target Feature for Speech Enhancementcited by 6
  8. 2022MTI-Net: A Multi-Target Speech Intelligibility Prediction Modelcited by 6
  9. 2022Exploring Self-Attention Mechanisms for Speech Separationcited by 4
  10. 2021Deep Sound Field Reconstruction in Real Rooms: Introducing the ISOBEL Sound Field Datasetcited by 3
  11. 2023Single-channel speech enhancement using learnable loss mixupcited by 2
  12. 2024Density Adaptive Attention is All You Need: Robust Parameter-Efficient Fine-Tuning Across Multiple Modalitiescited by 2
  13. 2023Multi-Task Pseudo-Label Learning for Non-Intrusive Speech Quality Assessment Modelcited by 1
  14. 2023Toward a Deeper Understanding: RetNet Viewed through Convolutioncited by 1
  15. 2024Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancementcited by 1
  16. 2024Selective State Space Model for Monaural Speech Enhancementcited by 1
  17. 2021Speech Enhancement using Separable Polling Attention and Global Layer Normalization followed with PReLUcited by 0
  18. 2021Full Attention Bidirectional Deep Learning Structure for Single Channel Speech Enhancementcited by 0
  19. 2022Parallel Gated Neural Network With Attention Mechanism For Speech Enhancementcited by 0
  20. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  21. 2024Stage-Wise and Prior-Aware Neural Speech Phase Predictioncited by 0
  22. 2025Efficient Neural and Numerical Methods for High-Quality Online Speech Spectrogram Inversion via Gradient Theoremcited by 0