# Learning Speech Emotion Representations in the Quaternion Domain

arXiv: https://arxiv.org/abs/2204.02385 · HTML: https://mlsanity.com/paper/2204.02385/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 75 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [Deep Quaternion Networks](https://mlsanity.com/paper/1712.04604/) (2017)
- [Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems](https://mlsanity.com/paper/1912.09257/) (2019)
- [You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation](https://mlsanity.com/paper/2005.07157/) (2020)
- [Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks](https://mlsanity.com/paper/1906.04393/) (2019)
- [PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions](https://mlsanity.com/paper/2110.04176/) (2021)
- [Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/1806.07789/) (2018)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events](https://mlsanity.com/paper/1812.06811/) (2018)
- [Quaternion Generative Adversarial Networks](https://mlsanity.com/paper/2104.09630/) (2021)
- [Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation](https://mlsanity.com/paper/2102.01813/) (2021)
- [A Quaternion-Valued Variational Autoencoder](https://mlsanity.com/paper/2010.11647/) (2020)
- [Speech Emotion Recognition Using Quaternion Convolutional Neural Networks](https://mlsanity.com/paper/2111.00404/) (2021)
- [Adversarial Machine Learning And Speech Emotion Recognition: Utilizing Generative Adversarial Networks For Robustness](https://mlsanity.com/paper/1811.11402/) (2018)
- [Emotion Recognition From Speech With Recurrent Neural Networks](https://mlsanity.com/paper/1701.08071/) (2017)
- [Dual Quaternion Ambisonics Array for Six-Degree-of-Freedom Acoustic Representation](https://mlsanity.com/paper/2204.01851/) (2022)
- [Capturing Long-term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition](https://mlsanity.com/paper/1708.07050/) (2017)
- [Quaternion Neural Networks for Multi-channel Distant Speech Recognition](https://mlsanity.com/paper/2005.08566/) (2020)
- [Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition](https://mlsanity.com/paper/1910.13806/) (2019)
- [Hybrid Data Augmentation and Deep Attention-based Dilated Convolutional-Recurrent Neural Networks for Speech Emotion Recognition](https://mlsanity.com/paper/2109.09026/) (2021)
- [Multi-Window Data Augmentation Approach for Speech Emotion Recognition](https://mlsanity.com/paper/2010.09895/) (2020)

## Built on by (most cited first, 7 of 7)

- [Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks](https://mlsanity.com/paper/2401.02277/) (2024)
- [MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition](https://mlsanity.com/paper/2308.04025/) (2023)
- [Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition](https://mlsanity.com/paper/2310.00283/) (2023)
- [PHemoNet: A Multimodal Network for Physiological Signals](https://mlsanity.com/paper/2410.00010/) (2024)
- [Demystifying the Hypercomplex: Inductive Biases in Hypercomplex Deep Learning](https://mlsanity.com/paper/2405.07024/) (2024)
- [Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition](https://mlsanity.com/paper/2405.00307/) (2024)
- [Multi-scale Quaternion CNN and BiGRU with Cross Self-attention Feature Fusion for Fault Diagnosis of Bearing](https://mlsanity.com/paper/2405.16114/) (2024)

## Abstract (arXiv)

The modeling of human emotion expression in speech signals is an important, yet challenging task. The high resource demand of speech emotion recognition models, combined with the the general scarcity of emotion-labelled data are obstacles to the development and application of effective solutions in this field. In this paper, we present an approach to jointly circumvent these difficulties. Our method, named RH-emo, is a novel semi-supervised architecture aimed at extracting quaternion embeddings from real-valued monoaural spectrograms, enabling the use of quaternion-valued networks for speech emotion recognition tasks. RH-emo is a hybrid real/quaternion autoencoder network that consists of a real-valued encoder in parallel to a real-valued emotion classifier and a quaternion-valued decoder. On the one hand, the classifier permits to optimize each latent axis of the embeddings for the classification of a specific emotion-related characteristic: valence, arousal, dominance and overall emotion. On the other hand, the quaternion reconstruction enables the latent dimension to develop intra-channel correlations that are required for an effective representation as a quaternion entity. We test our approach on speech emotion recognition tasks using four popular datasets: Iemocap, Ravdess, EmoDb and Tess, comparing the performance of three well-established real-valued CNN architectures (AlexNet, ResNet-50, VGG) and their quaternion-valued equivalent fed with the embeddings created with RH-emo. We obtain a consistent improvement in the test accuracy for all datasets, while drastically reducing the resources' demand of models. Moreover, we performed additional experiments and ablation studies that confirm the effectiveness of our approach. The RH-emo repository is available at: https://github.com/ispamm/rhemo.