mlsanityData to 7 Oct 2026

Exploring Self-Attention Mechanisms for Speech Separation

arXiv 2202.02884 ↗First posted 6 Feb 2022eess.AS cs.LG cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Transformers have enabled impressive improvements in deep learning. They often outperform recurrent and convolutional models in many tasks while taking advantage of parallel processing. Recently, we proposed the SepFormer, which obtains state-of-the-art performance in speech separation with the WSJ0-2/3 Mix datasets. This paper studies in-depth Transformers for speech separation. In particular, we extend our previous findings on the SepFormer by providing results on more challenging noisy and noisy-reverberant datasets, such as LibriMix, WHAM!, and WHAMR!. Moreover, we extend our model to perform speech enhancement and provide experimental evidence on denoising and dereverberation tasks. Finally, we investigate, for the first time in speech separation, the use of efficient self-attention mechanisms such as Linformers, Lonformers, and ReFormers. We found that they reduce memory requirements significantly. For example, we show that the Reformer-based attention outperforms the popular Conv-TasNet model on the WSJ0-2Mix dataset while being faster at inference and comparable in terms of memory consumption.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 44 of its 59 references that are in the map. First posted on arXiv 6 Feb 2022.

Builds on

44 of 59 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2018SDR - half-baked or well done?cited by 420
32 more
  1. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  2. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  3. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  4. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  5. 2020Attention is All You Need in Speech Separationcited by 184
  6. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  7. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  8. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  9. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 133
  10. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  11. 2021MetricGAN+: An Improved Version of MetricGAN for Speech Enhancementcited by 114
  12. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  13. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  14. 2020Wavesplit: End-to-End Speech Separation by Speaker Clusteringcited by 74
  15. 2020Continuous speech separation: dataset and analysiscited by 73
  16. 2020Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separationcited by 67
  17. 2019WHAMR!: Noisy and Reverberant Single-Channel Speech Separationcited by 61
  18. 2020Sudo rm -rf: Efficient Networks for Universal Audio Source Separationcited by 50
  19. 2019PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Networkcited by 49
  20. 2020Voice Separation with an Unknown Number of Multiple Speakerscited by 46
  21. 2021Vision Transformers with Patch Diversificationcited by 45
  22. 2020Continuous Speech Separation with Conformercited by 36
  23. 2019T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancementcited by 34
  24. 2019Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separationcited by 20
  25. 2021Dual-branch Attention-In-Attention Transformer for single-channel speech enhancementcited by 18
  26. 2019Two-Step Sound Source Separation: Training on Learned Latent Targetscited by 17
  27. 2021DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancementcited by 16
  28. 2019FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networkscited by 15
  29. 2020Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformercited by 15
  30. 2018Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspectivecited by 13
  31. 2021TransMask: A Compact and Fast Speech Separation Model Based on Transformercited by 9
  32. 2020On Loss Functions and Recurrency Training for GAN-based Speech Enhancement Systemscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  2. 2024Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separationcited by 45
  3. 2022Resource-Efficient Separation Transformercited by 11
  4. 2024Towards Audio Codec-based Speech Separationcited by 4