# Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation

arXiv: https://arxiv.org/abs/1809.07454 · HTML: https://mlsanity.com/paper/1809.07454/

## Key facts

- Cited by 406 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2018.

## Built on by (most cited first, 30 of 406)

- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Noise Estimation for Generative Diffusion Models](https://mlsanity.com/paper/2104.02600/) (2021)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [Proactive Detection of Voice Cloning with Localized Watermarking](https://mlsanity.com/paper/2401.17264/) (2024)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation](https://mlsanity.com/paper/2403.18257/) (2024)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [An efficient encoder-decoder architecture with top-down attention for speech separation](https://mlsanity.com/paper/2209.15200/) (2022)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [Cold Diffusion for Speech Enhancement](https://mlsanity.com/paper/2211.02527/) (2022)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement](https://mlsanity.com/paper/2302.14748/) (2023)
- [FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement](https://mlsanity.com/paper/2203.12188/) (2022)
- [MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions](https://mlsanity.com/paper/2302.11824/) (2023)

## Abstract (arXiv)

Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time-frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time-frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully-convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions (masks) to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network (TCN) consisting of stacked 1-D dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time-frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time-frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications.