# Boosting Unknown-number Speaker Separation with Transformer Decoder-based Attractor

arXiv: https://arxiv.org/abs/2401.12473 · HTML: https://mlsanity.com/paper/2401.12473/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2024.

## Built on by (most cited first, 6 of 6)

- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation](https://mlsanity.com/paper/2602.23165/) (2026)
- [A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy References](https://mlsanity.com/paper/2508.14623/) (2025)
- [Multiple Choice Learning for Efficient Speech Separation with Many Speakers](https://mlsanity.com/paper/2411.18497/) (2024)
- [TISDiSS: A Training-Time and Inference-Time Scalable Framework for Discriminative Source Separation](https://mlsanity.com/paper/2509.15666/) (2025)
- [Speech Separation for Hearing-Impaired Children in the Classroom](https://mlsanity.com/paper/2511.07677/) (2025)

## Abstract (arXiv)

We propose a novel speech separation model designed to separate mixtures with an unknown number of speakers. The proposed model stacks 1) a dual-path processing block that can model spectro-temporal patterns, 2) a transformer decoder-based attractor (TDA) calculation module that can deal with an unknown number of speakers, and 3) triple-path processing blocks that can model inter-speaker relations. Given a fixed, small set of learned speaker queries and the mixture embedding produced by the dual-path blocks, TDA infers the relations of these queries and generates an attractor vector for each speaker. The estimated attractors are then combined with the mixture embedding by feature-wise linear modulation conditioning, creating a speaker dimension. The mixture embedding, conditioned with speaker information produced by TDA, is fed to the final triple-path blocks, which augment the dual-path blocks with an additional pathway dedicated to inter-speaker processing. The proposed approach outperforms the previous best reported in the literature, achieving 24.0 and 23.7 dB SI-SDR improvement (SI-SDRi) on WSJ0-2 and 3mix respectively, with a single model trained to separate 2- and 3-speaker mixtures. The proposed model also exhibits strong performance and generalizability at counting sources and separating mixtures with up to 5 speakers.