# When Forgetting Looks Like Improvement: Metric Masking in Streaming Diarizer Adaptation and the Price of Rehearsal

arXiv: https://arxiv.org/abs/2610.08828 · HTML: https://mlsanity.com/paper/2610.08828/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments](https://mlsanity.com/paper/2107.04174/) (2021)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering](https://mlsanity.com/paper/2507.18446/) (2025)
- [Dissecting the Segmentation Model of End-to-End Diarization with Vector Clustering](https://mlsanity.com/paper/2506.11605/) (2025)
- [Benchmarking Diarization Models](https://mlsanity.com/paper/2509.26177/) (2025)

## Abstract (arXiv)

Small-data adaptation can improve speech detection while degrading speaker attribution. We study this discrepancy in a released streaming diarizer adapted on 7.5 h of two-party conversation and evaluated across six corpora. Adaptation substantially improves in-domain diarization performance and transfers to an independent corpus. However, this improvement is not consistent across evaluation scenarios as the additional confusion is mainly associated with impaired temporal identity consistency rather than speaker-count errors. A local-remapping diagnostic reveals different patterns of identity degradation across corpora, indicating that adaptation may alter how streaming models maintain speaker assignments over time. Rehearsal reduces the observed degradation but reduces the cross-domain transfer performance. These results highlight the need to jointly evaluate detection accuracy, identity consistency, and retention behavior when adapting streaming diarization systems.