# Probabilistic Fusion and Calibration of Neural Speaker Diarization Models

arXiv: https://arxiv.org/abs/2511.22696 · HTML: https://mlsanity.com/paper/2511.22696/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [Leveraging Self-Supervised Learning for Speaker Diarization](https://mlsanity.com/paper/2409.09408/) (2024)
- [From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.00890/) (2022)
- [DOVER-Lap: A Method for Combining Overlap-aware Diarization Outputs](https://mlsanity.com/paper/2011.01997/) (2020)
- [Overlap-aware low-latency online speaker diarization based on end-to-end local segmentation](https://mlsanity.com/paper/2109.06483/) (2021)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration](https://mlsanity.com/paper/2408.02841/) (2024)
- [EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers](https://mlsanity.com/paper/2203.17068/) (2022)
- [Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis](https://mlsanity.com/paper/2211.10243/) (2022)
- [DOVER: A Method for Combining Diarization Outputs](https://mlsanity.com/paper/1909.08090/) (2019)
- [Attention-based Encoder-Decoder End-to-End Neural Diarization with Embedding Enhancer](https://mlsanity.com/paper/2309.06672/) (2023)
- [DiaPer: End-to-End Neural Diarization with Perceiver-Based Attractors](https://mlsanity.com/paper/2312.04324/) (2023)
- [Attention-based Encoder-Decoder Network for End-to-End Neural Speaker Diarization with Target Speaker Attractor](https://mlsanity.com/paper/2305.10704/) (2023)
- [ASR-Aware End-to-end Neural Diarization](https://mlsanity.com/paper/2202.01286/) (2022)
- [On the calibration of powerset speaker diarization models](https://mlsanity.com/paper/2409.15885/) (2024)

## Built on by (most cited first, 1 of 1)

- [Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization](https://mlsanity.com/paper/2609.39162/) (2026)

## Abstract (arXiv)

End-to-End Neural Diarization (EEND) systems produce frame-level probabilistic speaker activity estimates, yet since evaluation focuses primarily on Diarization Error Rate (DER), the reliability and calibration of these confidence scores have been largely neglected. When fusing multiple diarization systems, DOVER-Lap remains the only established approach, operating at the segment level with hard decisions. We propose working with continuous probability outputs, which enables more sophisticated fusion and calibration techniques that can leverage model uncertainty and complementary strengths across different architectures. This paper presents the first comprehensive framework for calibrating and fusing EEND models at the probability level. We investigate two output formulations (multilabel and powerset representations) and their impact on calibration and fusion effectiveness. Through extensive experiments on the CallHome two-speaker benchmark, we demonstrate that proper calibration provides substantial improvements even for individual models (up to 19% relative DER reduction), in some cases mitigating the absence of domain adaptation. We reveal that joint calibration in powerset space consistently outperforms independent per-speaker calibration, that fusion substantially improves over individual models, and that the Fuse-then-Calibrate ordering generally outperforms both calibrating before fusion and uncalibrated fusion while requiring calibration of only a single combined model. Our best configuration outperforms DOVER-Lap in terms of DER while providing reliable confidence estimates essential for downstream applications. This work proposes best practices for probability-level fusion of EEND systems and demonstrates the advantages of leveraging soft outputs over hard decisions.