# A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions

arXiv: https://arxiv.org/abs/2401.00536 · HTML: https://mlsanity.com/paper/2401.00536/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 31 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information](https://mlsanity.com/paper/2203.15326/) (2022)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [Light-SERNet: A lightweight fully convolutional neural network for speech emotion recognition](https://mlsanity.com/paper/2110.03435/) (2021)
- [End-to-End Speech Emotion Recognition: Challenges of Real-Life Emergency Call Centers Data Recordings](https://mlsanity.com/paper/2110.14957/) (2021)
- [Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP](https://mlsanity.com/paper/2304.00860/) (2023)
- [End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations](https://mlsanity.com/paper/2310.02281/) (2023)
- [Exploring Attention Mechanisms for Multimodal Emotion Recognition in an Emergency Call Center Corpus](https://mlsanity.com/paper/2306.07115/) (2023)
- [Training speech emotion classifier without categorical annotations](https://mlsanity.com/paper/2210.07642/) (2022)
- [Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition](https://mlsanity.com/paper/2210.16642/) (2022)

## Built on by (most cited first, 1 of 1)

- [Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue](https://mlsanity.com/paper/2609.39072/) (2026)

## Abstract (arXiv)

Speech emotion recognition (SER) has received a great deal of attention in recent years in the context of spontaneous conversations. While there have been notable results on datasets like the well known corpus of naturalistic dyadic conversations, IEMOCAP, for both the case of categorical and dimensional emotions, there are few papers which try to predict both paradigms at the same time. Therefore, in this work, we aim to highlight the performance contribution of multi-task learning by proposing a multi-task, multi-modal system that predicts categorical and dimensional emotions. The results emphasise the importance of cross-regularisation between the two types of emotions. Our approach consists of a multi-task, multi-modal architecture that uses parallel feature refinement through self-attention for the feature of each modality. In order to fuse the features, our model introduces a set of learnable bridge tokens that merge the acoustic and linguistic features with the help of cross-attention. Our experiments for categorical emotions on 10-fold validation yield results comparable to the current state-of-the-art. In our configuration, our multi-task approach provides better results compared to learning each paradigm separately. On top of that, our best performing model achieves a high result for valence compared to the previous multi-task experiments.