# Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information

arXiv: https://arxiv.org/abs/2203.15326 · HTML: https://mlsanity.com/paper/2203.15326/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 5 of 5)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Emotion recognition by fusing time synchronous and time asynchronous representations](https://mlsanity.com/paper/2010.14102/) (2020)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Attentive Modality Hopping Mechanism for Speech Emotion Recognition](https://mlsanity.com/paper/1912.00846/) (2019)
- [Multi-Time-Scale Convolution for Emotion Recognition from Speech Audio Signals](https://mlsanity.com/paper/2003.03375/) (2020)

## Built on by (most cited first, 30 of 32)

- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning](https://mlsanity.com/paper/2305.09299/) (2023)
- [Disentangling Prosody Representations with Unsupervised Speech Reconstruction](https://mlsanity.com/paper/2212.06972/) (2022)
- [DWFormer: Dynamic Window transFormer for Speech Emotion Recognition](https://mlsanity.com/paper/2303.01694/) (2023)
- [Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP](https://mlsanity.com/paper/2304.00860/) (2023)
- [MFHCA: Enhancing Speech Emotion Recognition Via Multi-Spatial Fusion and Hierarchical Cooperative Attention](https://mlsanity.com/paper/2404.13509/) (2024)
- [Learning Robust Self-attention Features for Speech Emotion Recognition with Label-adaptive Mixup](https://mlsanity.com/paper/2305.06273/) (2023)
- [Knowledge Transfer For On-Device Speech Emotion Recognition with Neural Structured Learning](https://mlsanity.com/paper/2210.14977/) (2022)
- [Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations](https://mlsanity.com/paper/2309.04849/) (2023)
- [Test-Time Training for Speech](https://mlsanity.com/paper/2309.10930/) (2023)
- [Improving Speech Emotion Recognition with Unsupervised Speaking Style Transfer](https://mlsanity.com/paper/2211.08843/) (2022)
- [Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting](https://mlsanity.com/paper/2508.02429/) (2025)
- [Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.13340/) (2026)
- [Distribution-based Emotion Recognition in Conversation](https://mlsanity.com/paper/2211.04834/) (2022)
- [Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning](https://mlsanity.com/paper/2406.08931/) (2024)
- [Two-stage Framework for Robust Speech Emotion Recognition Using Target Speaker Extraction in Human Speech Noise Conditions](https://mlsanity.com/paper/2409.19585/) (2024)
- [hierarchical network with decoupled knowledge distillation for speech emotion recognition](https://mlsanity.com/paper/2303.05134/) (2023)
- [Evaluating raw waveforms with deep learning frameworks for speech emotion recognition](https://mlsanity.com/paper/2307.02820/) (2023)
- [A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions](https://mlsanity.com/paper/2401.00536/) (2023)
- [Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations](https://mlsanity.com/paper/2503.22687/) (2025)
- [Balancing Task-invariant Interaction and Task-specific Adaptation for Unified Image Fusion](https://mlsanity.com/paper/2504.05164/) (2025)
- [PTS-SNN: A Prompt-Tuned Temporal Shift Spiking Neural Networks for Efficient Speech Emotion Recognition](https://mlsanity.com/paper/2602.08240/) (2026)
- [MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition](https://mlsanity.com/paper/2306.09361/) (2023)
- [CochCeps-Augment: A Novel Self-Supervised Contrastive Learning Using Cochlear Cepstrum-based Masking for Speech Emotion Recognition](https://mlsanity.com/paper/2402.06923/) (2024)
- [Beyond the Labels: Unveiling Text-Dependency in Paralinguistic Speech Recognition Datasets](https://mlsanity.com/paper/2403.07767/) (2024)
- [emoDARTS: Joint Optimisation of CNN & Sequential Neural Network Architectures for Superior Speech Emotion Recognition](https://mlsanity.com/paper/2403.14083/) (2024)
- [Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages](https://mlsanity.com/paper/2503.21806/) (2025)
- [HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modeling](https://mlsanity.com/paper/2509.18570/) (2025)
- [Uncertainty-Aware 3D Emotional Talking Face Synthesis with Emotion Prior Distillation](https://mlsanity.com/paper/2601.19112/) (2026)

## Abstract (arXiv)

Speech Emotion Recognition (SER) aims to help the machine to understand human's subjective emotion from only audio information. However, extracting and utilizing comprehensive in-depth audio information is still a challenging task. In this paper, we propose an end-to-end speech emotion recognition system using multi-level acoustic information with a newly designed co-attention module. We firstly extract multi-level acoustic information, including MFCC, spectrogram, and the embedded high-level acoustic information with CNN, BiLSTM and wav2vec2, respectively. Then these extracted features are treated as multimodal inputs and fused by the proposed co-attention mechanism. Experiments are carried on the IEMOCAP dataset, and our model achieves competitive performance with two different speaker-independent cross-validation strategies. Our code is available on GitHub.