mlsanityData to 14 Oct 2026

Powerset multi-class cross entropy loss for neural speaker diarization

arXiv 2310.13025 ↗First posted 19 Oct 2023cs.SD cs.AI cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

Since its introduction in 2019, the whole end-to-end neural diarization (EEND) line of work has been addressing speaker diarization as a frame-wise multi-label classification problem with permutation-invariant training. Despite EEND showing great promise, a few recent works took a step back and studied the possible combination of (local) supervised EEND diarization with (global) unsupervised clustering. Yet, these hybrid contributions did not question the original multi-label formulation. We propose to switch from multi-label (where any two speakers can be active at the same time) to powerset multi-class classification (where dedicated classes are assigned to pairs of overlapping speakers). Through extensive experiments on 9 different benchmarks, we show that this formulation leads to significantly better performance (mostly on overlapping speech) and robustness to domain mismatch, while eliminating the detection threshold hyperparameter, critical for the multi-label formulation.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

19 of 25 references.

References · 19

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  3. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  4. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
  5. 2018Speaker Recognition from Raw Waveform with SincNetcited by 166
  6. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 88
  7. 2021End-to-end speaker segmentation for overlap-aware resegmentationcited by 80
  8. 2020Spot the conversation: speaker diarisation in the wildcited by 79
  9. 2019End-to-End Neural Speaker Diarization with Self-attentioncited by 77
  10. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 73
  11. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 71
  12. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 71
7 more
  1. 2021AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenariocited by 60
  2. 2020Integrating end-to-end neural and clustering-based diarization: Getting the best of both worldscited by 35
  3. 2019Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detectioncited by 27
  4. 2021Advances in integration of end-to-end neural and clustering-based diarization for real conversational speechcited by 23
  5. 2022From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarizationcited by 20
  6. 2022GPU-accelerated Guided Source Separation for Meeting Transcriptioncited by 10
  7. 2022Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysiscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 34; 2025: 33; 2026: 42.

111 papers.

Sort:

By area:

  1. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  2. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 122
  3. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  4. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 80
  5. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  6. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 36
  7. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  8. 2024Empirical evidence of Large Language Model's influence on human spoken communicationcited by 27
  9. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 23
  10. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 20
  11. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 20
  12. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 17
38 more
  1. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  2. 2024DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localizationcited by 12
  3. 2025BLAB: Brutally Long Audio Benchcited by 11
  4. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  5. 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
  6. 2025Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Modelscited by 7
  7. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  8. 2025SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generationcited by 6
  9. 2025Lotus: Creating Short Videos From Long Videos With Abstractive and Extractive Summarizationcited by 6
  10. 2024Deepfake Audio Detection Using Spectrogram-based Feature and Ensemble of Deep Learning Modelscited by 5
  11. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 5
  12. 2026TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Groundingcited by 5
  13. 2024Exploring Speech Foundation Models for Speaker Diarization in Child-Adult Dyadic Interactionscited by 4
  14. 2024The Greek podcast corpus: Competitive speech models for low-resourced languages with weakly supervised datacited by 4
  15. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 4
  16. 20241M-Deepfakes Detection Challengecited by 4
  17. 2024Mamba-based Segmentation Model for Speaker Diarizationcited by 4
  18. 2024Classification of Spontaneous and Scripted Speech for Multilingual Audiocited by 4
  19. 2025Voice of a Continent: Mapping Africa's Speech Technology Frontiercited by 4
  20. 2025CASPER: A Large Scale Spontaneous Speech Datasetcited by 4
  21. 2024Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversationcited by 3
  22. 2024Reverb: Open-Source ASR and Diarization from Revcited by 3
  23. 2025SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniorscited by 3
  24. 2025Challenges in Automated Processing of Speech from Child Wearables: The Case of Voice Type Classifiercited by 3
  25. 2025SDBench: A Comprehensive Benchmark Suite for Speaker Diarizationcited by 3
  26. 2025MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidancecited by 3
  27. 2026F-Actor: Controllable Conversational Behaviour in Full-Duplex Modelscited by 3
  28. 2024Listen Then See: Video Alignment with Speaker Attentioncited by 2
  29. 2024TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASRcited by 2
  30. 2024J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modelingcited by 2
  31. 2024ASR Benchmarking: Need for a More Representative Conversational Datasetcited by 2
  32. 2025Annotation Tool and Dataset for Fact-Checking Podcastscited by 2
  33. 2025Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modelingcited by 2
  34. 2025Benchmarking Diarization Modelscited by 2
  35. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
  36. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  37. 2024End-to-End Supervised Hierarchical Graph Clustering for Speaker Diarizationcited by 1
  38. 2024Joint vs Sequential Speaker-Role Detection and Automatic Speech Recognition for Air-traffic Controlcited by 1

50 most cited of 111 All 111 (JSON)