# Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention

arXiv: https://arxiv.org/abs/2512.04551 · HTML: https://mlsanity.com/paper/2512.04551/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 11 of 11)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [Self-attention fusion for audiovisual emotion recognition with incomplete data](https://mlsanity.com/paper/2201.11095/) (2022)
- [Scratching Visual Transformer's Back with Uniform Attention](https://mlsanity.com/paper/2210.08457/) (2022)
- [Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation](https://mlsanity.com/paper/2102.01813/) (2021)
- [A vector quantized masked autoencoder for speech emotion recognition](https://mlsanity.com/paper/2304.11117/) (2023)
- [Learning Robust Self-attention Features for Speech Emotion Recognition with Label-adaptive Mixup](https://mlsanity.com/paper/2305.06273/) (2023)
- [Speech Emotion Recognition Via CNN-Transformer and Multidimensional Attention Mechanism](https://mlsanity.com/paper/2403.04743/) (2024)

## Abstract (arXiv)

Speech emotion recognition (SER) is an important technology in human-computer interaction. However, achieving high performance is challenging due to emotional complexity and scarce annotated data. To tackle these challenges, we propose a multi-loss learning (MLL) framework integrating an energy-adaptive mixup (EAM) method and a frame-level attention module (FLAM). The EAM method leverages SNR-based augmentation to generate diverse speech samples capturing subtle emotional variations. FLAM enhances frame-level feature extraction for multi-frame emotional cues. Our MLL strategy combines Kullback-Leibler divergence, focal, center, and supervised contrastive loss to optimize learning, address class imbalance, and improve feature separability. We evaluate our method on four widely used SER datasets: IEMOCAP, MSP-IMPROV, RAVDESS, and SAVEE. The results demonstrate our method achieves state-of-the-art performance, suggesting its effectiveness and robustness.