# Learning Alignment for Multimodal Emotion Recognition from Speech

arXiv: https://arxiv.org/abs/1909.05645 · HTML: https://mlsanity.com/paper/1909.05645/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2019.

## Built on by (most cited first, 22 of 22)

- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification](https://mlsanity.com/paper/2102.07370/) (2021)
- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)
- [Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data](https://mlsanity.com/paper/2204.04645/) (2022)
- [Self-adaptive Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.11321/) (2024)
- [Pretrained audio neural networks for Speech emotion recognition in Portuguese](https://mlsanity.com/paper/2210.14716/) (2022)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [Multilevel Transformer For Multimodal Emotion Recognition](https://mlsanity.com/paper/2211.07711/) (2022)
- [Turbo your multi-modal classification with contrastive learning](https://mlsanity.com/paper/2409.09282/) (2024)
- [Multi-interaction TTS toward professional recording reproduction](https://mlsanity.com/paper/2507.00808/) (2025)
- [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers](https://mlsanity.com/paper/2306.13804/) (2023)
- [Topic Classification on Spoken Documents Using Deep Acoustic and Linguistic Features](https://mlsanity.com/paper/2106.08637/) (2021)
- [Audio-Visual Scene Classification Using A Transfer Learning Based Joint Optimization Strategy](https://mlsanity.com/paper/2204.11420/) (2022)
- [An Empirical Study and Improvement for Speech Emotion Recognition](https://mlsanity.com/paper/2304.03899/) (2023)
- [Fusion approaches for emotion recognition from speech using acoustic and text-based features](https://mlsanity.com/paper/2403.18635/) (2024)
- [Multi-modal Transfer Learning for Dynamic Facial Emotion Recognition in the Wild](https://mlsanity.com/paper/2504.21248/) (2025)

## Abstract (arXiv)

Speech emotion recognition is a challenging problem because human convey emotions in subtle and complex ways. For emotion recognition on human speech, one can either extract emotion related features from audio signals or employ speech recognition techniques to generate text from speech and then apply natural language processing to analyze the sentiment. Further, emotion recognition will be beneficial from using audio-textual multimodal information, it is not trivial to build a system to learn from multimodality. One can build models for two input sources separately and combine them in a decision level, but this method ignores the interaction between speech and text in the temporal domain. In this paper, we propose to use an attention mechanism to learn the alignment between speech frames and text words, aiming to produce more accurate multimodal feature representations. The aligned multimodal features are fed into a sequential model for emotion recognition. We evaluate the approach on the IEMOCAP dataset and the experimental results show the proposed approach achieves the state-of-the-art performance on the dataset.