# Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Models

arXiv: https://arxiv.org/abs/2506.18623 · HTML: https://mlsanity.com/paper/2506.18623/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks](https://mlsanity.com/paper/2012.14952/) (2020)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Dataset](https://mlsanity.com/paper/2203.16844/) (2022)
- [DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models](https://mlsanity.com/paper/2305.17651/) (2023)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models](https://mlsanity.com/paper/2207.06867/) (2022)
- [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023)

## Built on by (most cited first, 7 of 7)

- [MOSS-TTS Technical Report](https://mlsanity.com/paper/2603.18090/) (2026)
- [Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning](https://mlsanity.com/paper/2510.04213/) (2025)
- [MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions](https://mlsanity.com/paper/2603.28086/) (2026)
- [Benchmarking Diarization Models](https://mlsanity.com/paper/2509.26177/) (2025)
- [TCG CREST System Description for the DISPLACE-M Challenge](https://mlsanity.com/paper/2603.02030/) (2026)
- [Efficiency-Performance Trade-offs in Neural Speaker Diarization via Structured Pruning and Low-Bit Quantization](https://mlsanity.com/paper/2606.14030/) (2026)
- [Anuvani-Diarization: Zero Shot Speaker Diarization and Annotation Conventions Across 22 Languages](https://mlsanity.com/paper/2610.20327/) (2026)

## Abstract (arXiv)

This work presents a framework for compressing self-supervised models for speaker diarization through structured pruning guided by knowledge distillation. We investigate pruning objectives that target reducing both model parameters and computational complexity, where knowledge distillation enables compact models and pruning removes unnecessary parameters to improve hardware efficiency. We further analyze alternative pruning strategies, showing that a simple overall pruning approach provides the best balance between efficiency and accuracy. Compared to the original unpruned model, our method achieves up to 80% model size reduction and 4x faster inference without performance degradation. Comprehensive experiments across eight public diarization datasets demonstrate that the pruned models consistently match or surpass the performance of their uncompressed counterparts. Furthermore, we show strong outof-domain generalization on the CHiME-6 dataset, achieving accuracy comparable to the top systems in the CHiME-7 challenge without any domain adaptation. These results highlight that structured pruning, when guided by distillation, can yield efficient and generalizable diarization systems suitable for realworld applications.