# SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling

arXiv: https://arxiv.org/abs/2407.01857 · HTML: https://mlsanity.com/paper/2407.01857/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2024.

## Built on by (most cited first, 5 of 5)

- [LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Models](https://mlsanity.com/paper/2504.07402/) (2025)
- [TF-MLPNet: Tiny Real-Time Neural Speech Separation](https://mlsanity.com/paper/2508.03047/) (2025)
- [Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling](https://mlsanity.com/paper/2508.06393/) (2025)
- [Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching](https://mlsanity.com/paper/2603.12837/) (2026)
- [VorTEX: Various overlap ratio for Target speech EXtraction](https://mlsanity.com/paper/2603.14803/) (2026)

## Abstract (arXiv)

Real-time target speaker extraction (TSE) is intended to extract the desired speaker's voice from the observed mixture of multiple speakers in a streaming manner. Implementing real-time TSE is challenging as the computational complexity must be reduced to provide real-time operation. This work introduces to Conv-TasNet-based TSE a new architecture based on state space modeling (SSM) that has been shown to model long-term dependency effectively. Owing to SSM, fewer dilated convolutional layers are required to capture temporal dependency in Conv-TasNet, resulting in the reduction of model complexity. We also enlarge the window length and shift of the convolutional (TasNet) frontend encoder to reduce the computational cost further; the performance decline is compensated by over-parameterization of the frontend encoder. The proposed method reduces the real-time factor by 78% from the conventional causal Conv-TasNet-based TSE while matching its performance.