# Learning Emotion-Invariant Speaker Representations for Speaker Verification

arXiv: https://arxiv.org/abs/2505.18498 · HTML: https://mlsanity.com/paper/2505.18498/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Emotional speech synthesis and recognition, one of 11 directions in Speech and audio language models.
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- Speech and audio language models / [Emotional speech synthesis and recognition](https://mlsanity.com/speech/emotional-speech-synthesis-and-recognition/)

## Builds on (most cited first, 14 of 14)

- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition](https://mlsanity.com/paper/1906.07317/) (2019)
- [x-vectors meet emotions: A study on dependencies between emotion and speaker recognition](https://mlsanity.com/paper/2002.05039/) (2020)
- [Attention Back-end for Automatic Speaker Verification with Multiple Enrollment Utterances](https://mlsanity.com/paper/2104.01541/) (2021)
- [Multi-query multi-head attention pooling and Inter-topK penalty for speaker verification](https://mlsanity.com/paper/2110.05042/) (2021)
- [Exploring Binary Classification Loss For Speaker Verification](https://mlsanity.com/paper/2307.08205/) (2023)
- [Scoring of Large-Margin Embeddings for Speaker Verification: Cosine or PLDA?](https://mlsanity.com/paper/2204.03965/) (2022)
- [NPLDA: A Deep Neural PLDA Model for Speaker Verification](https://mlsanity.com/paper/2002.03562/) (2020)
- [CopyPaste: An Augmentation Method for Speech Emotion Recognition](https://mlsanity.com/paper/2010.14602/) (2020)
- [Probabilistic Spherical Discriminant Analysis: An Alternative to PLDA for length-normalized embeddings](https://mlsanity.com/paper/2203.14893/) (2022)

## Abstract (arXiv)

In recent years, the rapid progress in speaker verification (SV) technology has been driven by the extraction of speaker representations based on deep learning. However, such representations are still vulnerable to emotion variability. To address this issue, we propose multiple improvements to train speaker encoders to increase emotion robustness. Firstly, we utilize CopyPaste-based data augmentation to gather additional parallel data, which includes different emotional expressions from the same speaker. Secondly, we apply cosine similarity loss to restrict parallel sample pairs and minimize intra-class variation of speaker representations to reduce their correlation with emotional information. Finally, we use emotion-aware masking (EM) based on the speech signal energy on the input parallel samples to further strengthen the speaker representation and make it emotion-invariant. We conduct a comprehensive ablation study to demonstrate the effectiveness of these various components. Experimental results show that our proposed method achieves a relative 19.29% drop in EER compared to the baseline system.