# From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization

arXiv: https://arxiv.org/abs/2204.00890 · HTML: https://mlsanity.com/paper/2204.00890/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2022.

## Built on by (most cited first, 20 of 20)

- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models](https://mlsanity.com/paper/2508.06372/) (2025)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [In search of strong embedding extractors for speaker diarisation](https://mlsanity.com/paper/2210.14682/) (2022)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization](https://mlsanity.com/paper/2409.00819/) (2024)
- [Neural Diarization with Non-autoregressive Intermediate Attractors](https://mlsanity.com/paper/2303.06806/) (2023)
- [DiaCorrect: Error Correction Back-end For Speaker Diarization](https://mlsanity.com/paper/2309.08377/) (2023)
- [Speaker-Aware Simulation Improves Conversational Speech Recognition](https://mlsanity.com/paper/2602.04776/) (2026)
- [Flow-TSVAD: Target-Speaker Voice Activity Detection via Latent Flow Matching](https://mlsanity.com/paper/2409.04859/) (2024)
- [LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization](https://mlsanity.com/paper/2510.23320/) (2025)
- [Probabilistic Fusion and Calibration of Neural Speaker Diarization Models](https://mlsanity.com/paper/2511.22696/) (2025)
- [Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework](https://mlsanity.com/paper/2406.16315/) (2024)
- [Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios](https://mlsanity.com/paper/2407.01317/) (2024)
- [Improving Neural Diarization through Speaker Attribute Attractors and Local Dependency Modeling](https://mlsanity.com/paper/2506.05593/) (2025)
- [O-EENC-SD: Efficient Online End-to-End Neural Clustering for Speaker Diarization](https://mlsanity.com/paper/2512.15229/) (2025)
- [Efficient ASR Training with Conversations that Never Happened](https://mlsanity.com/paper/2606.03957/) (2026)
- [On the Role of Conversational Timing in Synthetic Training Data for ASR](https://mlsanity.com/paper/2607.08371/) (2026)
- [A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversations](https://mlsanity.com/paper/2609.28806/) (2026)

## Abstract (arXiv)

End-to-end neural diarization (EEND) is nowadays one of the most prominent research topics in speaker diarization. EEND presents an attractive alternative to standard cascaded diarization systems since a single system is trained at once to deal with the whole diarization problem. Several EEND variants and approaches are being proposed, however, all these models require large amounts of annotated data for training but available annotated data are scarce. Thus, EEND works have used mostly simulated mixtures for training. However, simulated mixtures do not resemble real conversations in many aspects. In this work we present an alternative method for creating synthetic conversations that resemble real ones by using statistics about distributions of pauses and overlaps estimated on genuine conversations. Furthermore, we analyze the effect of the source of the statistics, different augmentations and amounts of data. We demonstrate that our approach performs substantially better than the original one, while reducing the dependence on the fine-tuning stage. Experiments are carried out on 2-speaker telephone conversations of Callhome and DIHARD 3. Together with this publication, we release our implementations of EEND and the method for creating simulated conversations.