# Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention

arXiv: https://arxiv.org/abs/2106.04133 · HTML: https://mlsanity.com/paper/2106.04133/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 6 of 6)

- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://mlsanity.com/paper/1805.09843/) (2018)
- [Learning Alignment for Multimodal Emotion Recognition from Speech](https://mlsanity.com/paper/1909.05645/) (2019)
- [Speech Emotion Recognition Using Multi-hop Attention Mechanism](https://mlsanity.com/paper/1904.10788/) (2019)
- [Deep neural networks for emotion recognition combining audio and transcripts](https://mlsanity.com/paper/1911.00432/) (2019)

## Built on by (most cited first, 14 of 14)

- [Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information](https://mlsanity.com/paper/2203.15326/) (2022)
- [Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition](https://mlsanity.com/paper/2211.08233/) (2022)
- [Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP](https://mlsanity.com/paper/2304.00860/) (2023)
- [Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment](https://mlsanity.com/paper/2412.20821/) (2024)
- [SpeechEQ: Speech Emotion Recognition based on Multi-scale Unified Datasets and Multitask Learning](https://mlsanity.com/paper/2206.13101/) (2022)
- [MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition](https://mlsanity.com/paper/2308.04025/) (2023)
- [Improving Speech Emotion Recognition Through Focus and Calibration Attention Mechanisms](https://mlsanity.com/paper/2208.10491/) (2022)
- [Pretrained audio neural networks for Speech emotion recognition in Portuguese](https://mlsanity.com/paper/2210.14716/) (2022)
- [Multi-Convformer: Extending Conformer with Multiple Convolution Kernels](https://mlsanity.com/paper/2407.03718/) (2024)
- [hierarchical network with decoupled knowledge distillation for speech emotion recognition](https://mlsanity.com/paper/2303.05134/) (2023)
- [Self context-aware emotion perception on human-robot interaction](https://mlsanity.com/paper/2401.10946/) (2024)
- [Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations](https://mlsanity.com/paper/2503.22687/) (2025)
- [Enhancing Speech Emotion Recognition through Segmental Average Pooling of Self-Supervised Learning Features](https://mlsanity.com/paper/2410.12416/) (2024)
- [Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research](https://mlsanity.com/paper/2604.25776/) (2026)

## Abstract (arXiv)

Emotion recognition from speech is a challenging task. Re-cent advances in deep learning have led bi-directional recur-rent neural network (Bi-RNN) and attention mechanism as astandard method for speech emotion recognition, extractingand attending multi-modal features - audio and text, and thenfusing them for downstream emotion classification tasks. Inthis paper, we propose a simple yet efficient neural networkarchitecture to exploit both acoustic and lexical informationfrom speech. The proposed framework using multi-scale con-volutional layers (MSCNN) to obtain both audio and text hid-den representations. Then, a statistical pooling unit (SPU)is used to further extract the features in each modality. Be-sides, an attention module can be built on top of the MSCNN-SPU (audio) and MSCNN (text) to further improve the perfor-mance. Extensive experiments show that the proposed modeloutperforms previous state-of-the-art methods on IEMOCAPdataset with four emotion categories (i.e., angry, happy, sadand neutral) in both weighted accuracy (WA) and unweightedaccuracy (UA), with an improvement of 5.0% and 5.2% respectively under the ASR setting.