# A Stage-Wise Learning Strategy with Fixed Anchors for Robust Speaker Verification

arXiv: https://arxiv.org/abs/2510.18530 · HTML: https://mlsanity.com/paper/2510.18530/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 13 of 13)

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [VoiceID Loss: Speech Enhancement for Speaker Verification](https://mlsanity.com/paper/1904.03601/) (2019)
- [Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning](https://mlsanity.com/paper/2407.15188/) (2024)
- [Within-sample variability-invariant loss for robust speaker recognition under noisy environments](https://mlsanity.com/paper/2002.00924/) (2020)
- [Feature Enhancement with Deep Feature Losses for Speaker Verification](https://mlsanity.com/paper/1910.11905/) (2019)
- [Extended U-Net for Speaker Verification in Noisy Environments](https://mlsanity.com/paper/2206.13044/) (2022)
- [Diff-SV: A Unified Hierarchical Framework for Noise-Robust Speaker Verification Using Score-Based Diffusion Probabilistic Models](https://mlsanity.com/paper/2309.08320/) (2023)
- [A Joint Noise Disentanglement and Adversarial Training Framework for Robust Speaker Verification](https://mlsanity.com/paper/2408.11562/) (2024)

## Abstract (arXiv)

Learning robust speaker representations under noisy conditions presents significant challenges, which requires careful handling of both discriminative and noise-invariant properties. In this work, we proposed an anchor-based stage-wise learning strategy for robust speaker representation learning. Specifically, our approach begins by training a base model to establish discriminative speaker boundaries, and then extract anchor embeddings from this model as stable references. Finally, a copy of the base model is fine-tuned on noisy inputs, regularized by enforcing proximity to their corresponding fixed anchor embeddings to preserve speaker identity under distortion. Experimental results suggest that this strategy offers advantages over conventional joint optimization, particularly in maintaining discrimination while improving noise robustness. The proposed method demonstrates consistent improvements across various noise conditions, potentially due to its ability to handle boundary stabilization and variation suppression separately.