# Leveraging Self-Supervised Learning for Speaker Diarization

arXiv: https://arxiv.org/abs/2409.09408 · HTML: https://mlsanity.com/paper/2409.09408/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2024.

## Built on by (most cited first, 23 of 23)

- [MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction](https://mlsanity.com/paper/2604.27393/) (2026)
- [DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2602.12160/) (2026)
- [SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models](https://mlsanity.com/paper/2508.06372/) (2025)
- [MOSS-TTS Technical Report](https://mlsanity.com/paper/2603.18090/) (2026)
- [MOSS-Audio Technical Report](https://mlsanity.com/paper/2606.01802/) (2026)
- [Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Models](https://mlsanity.com/paper/2506.18623/) (2025)
- [MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions](https://mlsanity.com/paper/2603.28086/) (2026)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [Benchmarking Diarization Models](https://mlsanity.com/paper/2509.26177/) (2025)
- [Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition](https://mlsanity.com/paper/2510.03723/) (2025)
- [Probabilistic Fusion and Calibration of Neural Speaker Diarization Models](https://mlsanity.com/paper/2511.22696/) (2025)
- [MK-SGC-SC: Multiple Kernel Guided Sparse Graph Construction in Spectral Clustering for Unsupervised Speaker Diarization](https://mlsanity.com/paper/2601.19946/) (2026)
- [Make It Hard to Hear, Easy to Learn: Long-Form Bengali ASR and Speaker Diarization via Extreme Augmentation and Perfect Alignment](https://mlsanity.com/paper/2602.23070/) (2026)
- [TCG CREST System Description for the DISPLACE-M Challenge](https://mlsanity.com/paper/2603.02030/) (2026)
- [An Investigation Into Various Approaches For Bengali Long-Form Speech Transcription and Bengali Speaker Diarization](https://mlsanity.com/paper/2603.03158/) (2026)
- [Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics](https://mlsanity.com/paper/2603.22709/) (2026)
- ["OK Aura, Be Fair With Me": Demographics-Agnostic Training for Bias Mitigation in Wake-up Word Detection](https://mlsanity.com/paper/2604.05830/) (2026)
- [An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge](https://mlsanity.com/paper/2607.12468/) (2026)
- [Model-Agnostic and Language-Agnostic Voice Pipeline Improvement for the Agriculture Domain](https://mlsanity.com/paper/2609.20504/) (2026)
- [Transsion's Speaker-Attributed Multilingual ASR System for the MLC-SLM 2026 Challenge](https://mlsanity.com/paper/2609.20833/) (2026)
- [Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization](https://mlsanity.com/paper/2609.39162/) (2026)
- [Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings](https://mlsanity.com/paper/2609.39344/) (2026)
- [FASTDIAR: Frame-level speaker encoder for Streaming Diarization](https://mlsanity.com/paper/2610.02941/) (2026)

## Abstract (arXiv)

End-to-end neural diarization has evolved considerably over the past few years, but data scarcity is still a major obstacle for further improvements. Self-supervised learning methods such as WavLM have shown promising performance on several downstream tasks, but their application on speaker diarization is somehow limited. In this work, we explore using WavLM to alleviate the problem of data scarcity for neural diarization training. We use the same pipeline as Pyannote and improve the local end-to-end neural diarization with WavLM and Conformer. Experiments on far-field AMI, AISHELL-4, and AliMeeting datasets show that our method substantially outperforms the Pyannote baseline and achieves new state-of-the-art results on AMI and AISHELL-4, respectively. In addition, by analyzing the system performance under different data quantity scenarios, we show that WavLM representations are much more robust against data scarcity than filterbank features, enabling less data hungry training strategies. Furthermore, we found that simulated data, usually used to train endto-end diarization models, does not help when using WavLM in our experiments. Additionally, we also evaluate our model on the recent CHiME8 NOTSOFAR-1 task where it achieves better performance than the Pyannote baseline. Our source code is publicly available at https://github.com/BUTSpeechFIT/DiariZen.