# MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions

arXiv: https://arxiv.org/abs/2302.11824 · HTML: https://mlsanity.com/paper/2302.11824/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Transformer Quality in Linear Time](https://mlsanity.com/paper/2202.10447/) (2022)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [WHAMR!: Noisy and Reverberant Single-Channel Speech Separation](https://mlsanity.com/paper/1910.10279/) (2019)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020)
- [Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation](https://mlsanity.com/paper/1904.11148/) (2019)
- [Two-Step Sound Source Separation: Training on Learned Latent Targets](https://mlsanity.com/paper/1910.09804/) (2019)
- [Filterbank design for end-to-end speech separation](https://mlsanity.com/paper/1910.10400/) (2019)
- [SepIt: Approaching a Single Channel Speech Separation Bound](https://mlsanity.com/paper/2205.11801/) (2022)
- [Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective](https://mlsanity.com/paper/1811.09010/) (2018)

## Built on by (most cited first, 27 of 27)

- [Efficient Neural Music Generation](https://mlsanity.com/paper/2305.15719/) (2023)
- [Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation](https://mlsanity.com/paper/2403.18257/) (2024)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis](https://mlsanity.com/paper/2407.09732/) (2024)
- [Noise-robust Speech Separation with Fast Generative Correction](https://mlsanity.com/paper/2406.07461/) (2024)
- [SepMamba: State-space models for speaker separation using Mamba](https://mlsanity.com/paper/2410.20997/) (2024)
- [SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios](https://mlsanity.com/paper/2410.01481/) (2024)
- [A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy References](https://mlsanity.com/paper/2508.14623/) (2025)
- [On Time Domain Conformer Models for Monaural Speech Separation in Noisy Reverberant Acoustic Environments](https://mlsanity.com/paper/2310.06125/) (2023)
- [AG-LSEC: Audio Grounded Lexical Speaker Error Correction](https://mlsanity.com/paper/2406.17266/) (2024)
- [Improving Generalization of Speech Separation in Real-World Scenarios: Strategies in Simulation, Optimization, and Evaluation](https://mlsanity.com/paper/2408.16126/) (2024)
- [Multiple Choice Learning for Efficient Speech Separation with Many Speakers](https://mlsanity.com/paper/2411.18497/) (2024)
- [EDNet: A Versatile Speech Enhancement Framework with Gating Mamba Mechanism and Phase Shift-Invariant Training](https://mlsanity.com/paper/2506.16231/) (2025)
- [Knowing When to Quit: Probabilistic Early Exits for Speech Separation](https://mlsanity.com/paper/2507.09768/) (2025)
- [Blind Source Separation of Single-Channel Mixtures via Multi-Encoder Autoencoders](https://mlsanity.com/paper/2309.07138/) (2023)
- [Robustness of Speech Separation Models for Similar-pitch Speakers](https://mlsanity.com/paper/2407.15749/) (2024)
- [SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Models](https://mlsanity.com/paper/2501.08421/) (2025)
- [TF-MLPNet: Tiny Real-Time Neural Speech Separation](https://mlsanity.com/paper/2508.03047/) (2025)
- [When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection](https://mlsanity.com/paper/2608.30204/) (2026)
- [Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge](https://mlsanity.com/paper/2402.01413/) (2024)
- [Survey: Transformer-based Models in Data Modality Conversion](https://mlsanity.com/paper/2408.04723/) (2024)
- [Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation](https://mlsanity.com/paper/2504.02302/) (2025)
- [Passive Underwater Acoustic Signal Separation based on Feature Decoupling Dual-path Network](https://mlsanity.com/paper/2504.08371/) (2025)
- [Source Separation for A Cappella Music](https://mlsanity.com/paper/2509.26580/) (2025)
- [TFGA-Net: Temporal-Frequency Graph Attention Network for Brain-Controlled Speaker Extraction](https://mlsanity.com/paper/2510.12275/) (2025)
- [LP-CFM: Perceptual Invariance-Aware Conditional Flow Matching for Speech Modeling](https://mlsanity.com/paper/2512.20314/) (2025)
- [Cleaner Speech, Weaker Generalization: Revisiting Pitt-Derived Benchmarks for Alzheimer's Disease Detection](https://mlsanity.com/paper/2609.00276/) (2026)

## Abstract (arXiv)

Transformer based models have provided significant performance improvements in monaural speech separation. However, there is still a performance gap compared to a recent proposed upper bound. The major limitation of the current dual-path Transformer models is the inefficient modelling of long-range elemental interactions and local feature patterns. In this work, we achieve the upper bound by proposing a gated single-head transformer architecture with convolution-augmented joint self-attentions, named textitMossFormer (textitMonaural textitspeech textitseparation TranstextitFormer). To effectively solve the indirect elemental interactions across chunks in the dual-path architecture, MossFormer employs a joint local and global self-attention architecture that simultaneously performs a full-computation self-attention on local chunks and a linearised low-cost self-attention over the full sequence. The joint attention enables MossFormer model full-sequence elemental interaction directly. In addition, we employ a powerful attentive gating mechanism with simplified single-head self-attentions. Besides the attentive long-range modelling, we also augment MossFormer with convolutions for the position-wise local pattern modelling. As a consequence, MossFormer significantly outperforms the previous models and achieves the state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! benchmarks. Our model achieves the SI-SDRi upper bound of 21.2 dB on WSJ0-3mix and only 0.3 dB below the upper bound of 23.1 dB on WSJ0-2mix.