# Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation

arXiv: https://arxiv.org/abs/2103.00819 · HTML: https://mlsanity.com/paper/2103.00819/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding](https://mlsanity.com/paper/1709.04696/) (2017)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020)
- [Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation](https://mlsanity.com/paper/1904.11148/) (2019)
- [FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks](https://mlsanity.com/paper/1902.04891/) (2019)
- [End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction](https://mlsanity.com/paper/1804.10204/) (2018)
- [Mixup-breakdown: a consistency training method for improving generalization of speech separation models](https://mlsanity.com/paper/1910.13253/) (2019)
- [End-to-end training of time domain audio separation and recognition](https://mlsanity.com/paper/1912.08462/) (2019)
- [Phase-aware Single-stage Speech Denoising and Dereverberation with U-Net](https://mlsanity.com/paper/2006.00687/) (2020)

## Built on by (most cited first, 10 of 10)

- [Efficient Neural Music Generation](https://mlsanity.com/paper/2305.15719/) (2023)
- [An efficient encoder-decoder architecture with top-down attention for speech separation](https://mlsanity.com/paper/2209.15200/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning](https://mlsanity.com/paper/2303.03737/) (2023)
- [Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.04275/) (2021)
- [Papez: Resource-Efficient Speech Separation with Auditory Working Memory](https://mlsanity.com/paper/2407.00888/) (2024)
- [Progressive Learning for Stabilizing Label Selection in Speech Separation with Mapping-based Method](https://mlsanity.com/paper/2110.10593/) (2021)
- [Improving Label Assignments Learning by Dynamic Sample Dropout Combined with Layer-wise Optimization in Speech Separation](https://mlsanity.com/paper/2311.12199/) (2023)
- [Spectron: Target Speaker Extraction using Conditional Transformer with Adversarial Refinement](https://mlsanity.com/paper/2409.01352/) (2024)
- [Brainprint-Modulated Target Speaker Extraction](https://mlsanity.com/paper/2509.17883/) (2025)

## Abstract (arXiv)

One of the leading single-channel speech separation (SS) models is based on a TasNet with a dual-path segmentation technique, where the size of each segment remains unchanged throughout all layers. In contrast, our key finding is that multi-granularity features are essential for enhancing contextual modeling and computational efficiency. We introduce a self-attentive network with a novel sandglass-shape, namely Sandglasset, which advances the state-of-the-art (SOTA) SS performance at significantly smaller model size and computational cost. Forward along each block inside Sandglasset, the temporal granularity of the features gradually becomes coarser until reaching half of the network blocks, and then successively turns finer towards the raw signal level. We also unfold that residual connections between features with the same granularity are critical for preserving information after passing through the bottleneck layer. Experiments show our Sandglasset with only 2.3M parameters has achieved the best results on two benchmark SS datasets -- WSJ0-2mix and WSJ0-3mix, where the SI-SNRi scores have been improved by absolute 0.8 dB and 2.4 dB, respectively, comparing to the prior SOTA results.