# Unsupervised Personalization of an Emotion Recognition System: The Unique Properties of the Externalization of Valence in Speech

arXiv: https://arxiv.org/abs/2201.07876 · HTML: https://mlsanity.com/paper/2201.07876/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 65 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 4 of 4)

- [Progressive Neural Networks for Transfer Learning in Emotion Recognition](https://mlsanity.com/paper/1706.03256/) (2017)
- [Incorporating End-to-End Speech Recognition Models for Sentiment Analysis](https://mlsanity.com/paper/1902.11245/) (2019)
- [Ladder Networks for Emotion Recognition: Using Unsupervised Auxiliary Tasks to Improve Predictions of Emotional Attributes](https://mlsanity.com/paper/1804.10816/) (2018)
- [Identifying Mood Episodes Using Dialogue Features from Clinical Interviews](https://mlsanity.com/paper/1910.05115/) (2019)

## Built on by (most cited first, 7 of 7)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning](https://mlsanity.com/paper/2209.15449/) (2022)
- [Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models](https://mlsanity.com/paper/2305.19184/) (2023)
- [Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction](https://mlsanity.com/paper/2304.00377/) (2023)
- [Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction](https://mlsanity.com/paper/2206.06680/) (2022)
- [Enrolment-based personalisation for improving individual-level fairness in speech emotion recognition](https://mlsanity.com/paper/2406.06665/) (2024)
- [Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model](https://mlsanity.com/paper/2405.01730/) (2024)

## Abstract (arXiv)

The prediction of valence from speech is an important, but challenging problem. The externalization of valence in speech has speaker-dependent cues, which contribute to performances that are often significantly lower than the prediction of other emotional attributes such as arousal and dominance. A practical approach to improve valence prediction from speech is to adapt the models to the target speakers in the test set. Adapting a speech emotion recognition (SER) system to a particular speaker is a hard problem, especially with deep neural networks (DNNs), since it requires optimizing millions of parameters. This study proposes an unsupervised approach to address this problem by searching for speakers in the train set with similar acoustic patterns as the speaker in the test set. Speech samples from the selected speakers are used to create the adaptation set. This approach leverages transfer learning using pre-trained models, which are adapted with these speech samples. We propose three alternative adaptation strategies: unique speaker, oversampling and weighting approaches. These methods differ on the use of the adaptation set in the personalization of the valence models. The results demonstrate that a valence prediction model can be efficiently personalized with these unsupervised approaches, leading to relative improvements as high as 13.52%.