# SpeechEQ: Speech Emotion Recognition based on Multi-scale Unified Datasets and Multitask Learning

arXiv: https://arxiv.org/abs/2206.13101 · HTML: https://mlsanity.com/paper/2206.13101/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Towards Speaker Age Estimation with Label Distribution Learning](https://mlsanity.com/paper/2202.11424/) (2022)
- [Key-Sparse Transformer for Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2106.11532/) (2021)
- [Speech2Video: Cross-Modal Distillation for Speech to Video Generation](https://mlsanity.com/paper/2107.04806/) (2021)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [Variational Information Bottleneck for Effective Low-resource Audio Classification](https://mlsanity.com/paper/2107.04803/) (2021)

## Built on by (most cited first, 4 of 4)

- [SVLDL: Improved Speaker Age Estimation Using Selective Variance Label Distribution Learning](https://mlsanity.com/paper/2210.09524/) (2022)
- [Efficient Multi-Model Fusion with Adversarial Complementary Representation Learning](https://mlsanity.com/paper/2404.15704/) (2024)
- [Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition](https://mlsanity.com/paper/2602.01547/) (2026)
- [Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research](https://mlsanity.com/paper/2604.25776/) (2026)

## Abstract (arXiv)

Speech emotion recognition (SER) has many challenges, but one of the main challenges is that each framework does not have a unified standard. In this paper, we propose SpeechEQ, a framework for unifying SER tasks based on a multi-scale unified metric. This metric can be trained by Multitask Learning (MTL), which includes two emotion recognition tasks of Emotion States Category (EIS) and Emotion Intensity Scale (EIS), and two auxiliary tasks of phoneme recognition and gender recognition. For this framework, we build a Mandarin SER dataset - SpeechEQ Dataset (SEQD). We conducted experiments on the public CASIA and ESD datasets in Mandarin, which exhibit that our method outperforms baseline methods by a relatively large margin, yielding 8.0% and 6.5% improvement in accuracy respectively. Additional experiments on IEMOCAP with four emotion categories (i.e., angry, happy, sad, and neutral) also show the proposed method achieves a state-of-the-art of both weighted accuracy (WA) of 78.16% and unweighted accuracy (UA) of 77.47%.