# Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

arXiv: https://arxiv.org/abs/2609.04690 · HTML: https://mlsanity.com/paper/2609.04690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 6 of 6)

- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers](https://mlsanity.com/paper/2402.07327/) (2024)
- [A Novel Approach to for Multimodal Emotion Recognition : Multimodal semantic information fusion](https://mlsanity.com/paper/2502.08573/) (2025)
- [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers](https://mlsanity.com/paper/2306.13804/) (2023)
- [EmoTech: A Multi-modal Speech Emotion Recognition Using Multi-source Low-level Information with Hybrid Recurrent Network](https://mlsanity.com/paper/2501.12674/) (2025)

## Abstract (arXiv)

Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.