# x-vectors meet emotions: A study on dependencies between emotion and speaker recognition

arXiv: https://arxiv.org/abs/2002.05039 · HTML: https://mlsanity.com/paper/2002.05039/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 30 of 31)

- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation](https://mlsanity.com/paper/2108.02510/) (2021)
- [Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition](https://mlsanity.com/paper/2309.10294/) (2023)
- [Non-Contrastive Self-supervised Learning for Utterance-Level Information Extraction from Speech](https://mlsanity.com/paper/2208.05445/) (2022)
- [Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition](https://mlsanity.com/paper/2201.11826/) (2022)
- [Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors](https://mlsanity.com/paper/2203.04443/) (2022)
- [Revealing Emotional Clusters in Speaker Embeddings: A Contrastive Learning Strategy for Speech Emotion Recognition](https://mlsanity.com/paper/2401.11017/) (2024)
- [The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization](https://mlsanity.com/paper/2601.11846/) (2026)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [The Role of Phonetic Units in Speech Emotion Recognition](https://mlsanity.com/paper/2108.01132/) (2021)
- [Classification of Emotions and Evaluation of Customer Satisfaction from Speech in Real World Acoustic Environments](https://mlsanity.com/paper/2108.11981/) (2021)
- [Non-Contrastive Self-Supervised Learning of Utterance-Level Speech Representations](https://mlsanity.com/paper/2208.05413/) (2022)
- [A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2304.11472/) (2023)
- [Distribution-based Emotion Recognition in Conversation](https://mlsanity.com/paper/2211.04834/) (2022)
- [An Overview of Indian Spoken Language Recognition from Machine Learning Perspective](https://mlsanity.com/paper/2212.03812/) (2022)
- [Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement](https://mlsanity.com/paper/2401.04511/) (2024)
- [LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention](https://mlsanity.com/paper/2506.02083/) (2025)
- [NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion](https://mlsanity.com/paper/2511.00256/) (2025)
- [An Attribute-Aligned Strategy for Learning Speech Representation](https://mlsanity.com/paper/2106.02810/) (2021)
- [Beyond Isolated Utterances: Conversational Emotion Recognition](https://mlsanity.com/paper/2109.06112/) (2021)
- [Analysis of impact of emotions on target speech extraction and speech separation](https://mlsanity.com/paper/2208.07091/) (2022)
- [Training speech emotion classifier without categorical annotations](https://mlsanity.com/paper/2210.07642/) (2022)
- [On the Impact of Voice Anonymization on Speech Diagnostic Applications: a Case Study on COVID-19 Detection](https://mlsanity.com/paper/2304.02181/) (2023)
- [Evaluation of Speaker Anonymization on Emotional Speech](https://mlsanity.com/paper/2305.01759/) (2023)
- [Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?](https://mlsanity.com/paper/2402.01579/) (2024)
- [Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation](https://mlsanity.com/paper/2603.10827/) (2026)
- [Significance of Speaker Embeddings and Temporal Context for Depression Detection](https://mlsanity.com/paper/2107.13969/) (2021)
- [Nonwords Pronunciation Classification in Language Development Tests for Preschool Children](https://mlsanity.com/paper/2206.08058/) (2022)

## Abstract (arXiv)

In this work, we explore the dependencies between speaker recognition and emotion recognition. We first show that knowledge learned for speaker recognition can be reused for emotion recognition through transfer learning. Then, we show the effect of emotion on speaker recognition. For emotion recognition, we show that using a simple linear model is enough to obtain good performance on the features extracted from pre-trained models such as the x-vector model. Then, we improve emotion recognition performance by fine-tuning for emotion classification. We evaluated our experiments on three different types of datasets: IEMOCAP, MSP-Podcast, and Crema-D. By fine-tuning, we obtained 30.40%, 7.99%, and 8.61% absolute improvement on IEMOCAP, MSP-Podcast, and Crema-D respectively over baseline model with no pre-training. Finally, we present results on the effect of emotion on speaker verification. We observed that speaker verification performance is prone to changes in test speaker emotions. We found that trials with angry utterances performed worst in all three datasets. We hope our analysis will initiate a new line of research in the speaker recognition community.