# EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers

arXiv: https://arxiv.org/abs/2203.17068 · HTML: https://mlsanity.com/paper/2203.17068/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2022.

## Built on by (most cited first, 10 of 10)

- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition](https://mlsanity.com/paper/2310.01688/) (2023)
- [Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering](https://mlsanity.com/paper/2210.13690/) (2022)
- [Neural Blind Source Separation and Diarization for Distant Speech Recognition](https://mlsanity.com/paper/2406.08396/) (2024)
- [EmoDiarize: Speaker Diarization and Emotion Identification from Speech Signals using Convolutional Neural Networks](https://mlsanity.com/paper/2310.12851/) (2023)
- [Probabilistic Fusion and Calibration of Neural Speaker Diarization Models](https://mlsanity.com/paper/2511.22696/) (2025)
- [Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios](https://mlsanity.com/paper/2407.01317/) (2024)
- [Speaker Diarization for Low-Resource Languages Through Wav2vec Fine-Tuning](https://mlsanity.com/paper/2504.18582/) (2025)
- [Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space](https://mlsanity.com/paper/2606.01909/) (2026)
- [Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model](https://mlsanity.com/paper/2609.12154/) (2026)

## Abstract (arXiv)

In this paper, we present a novel framework that jointly performs three tasks: speaker diarization, speech separation, and speaker counting. Our proposed framework integrates speaker diarization based on end-to-end neural diarization (EEND) models, speaker counting with encoder-decoder based attractors (EDA), and speech separation using Conv-TasNet. In addition, we propose a multiple 1x1 convolutional layer architecture for estimating the separation masks corresponding to a flexible number of speakers and a fusion technique for refining the separated speech signal with obtained speaker diarization information to improve the joint framework. Experiments using the LibriMix dataset show that our proposed method outperforms the single-task baselines in both diarization and separation metrics for fixed and flexible numbers of speakers and improves speaker counting performance for flexible numbers of speakers. All materials will be open-sourced and reproducible in ESPnet toolkit.