# Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching

arXiv: https://arxiv.org/abs/2009.02598 · HTML: https://mlsanity.com/paper/2009.02598/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2020.

## Built on by (most cited first, 13 of 13)

- [DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction](https://mlsanity.com/paper/2110.12942/) (2021)
- [Towards Robust Multimodal Sentiment Analysis with Incomplete Data](https://mlsanity.com/paper/2409.20012/) (2024)
- [RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates](https://mlsanity.com/paper/2312.10386/) (2023)
- [Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data](https://mlsanity.com/paper/2202.00097/) (2022)
- [MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2111.00865/) (2021)
- [Noise-Resistant Multimodal Transformer for Emotion Recognition](https://mlsanity.com/paper/2305.02814/) (2023)
- [Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding](https://mlsanity.com/paper/2210.11933/) (2022)
- [SemiMemes: A Semi-supervised Learning Approach for Multimodal Memes Analysis](https://mlsanity.com/paper/2304.00020/) (2023)
- [Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation](https://mlsanity.com/paper/2302.01555/) (2023)
- [AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations](https://mlsanity.com/paper/2401.15164/) (2024)
- [GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints](https://mlsanity.com/paper/2506.00865/) (2025)
- [AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning](https://mlsanity.com/paper/2402.10921/) (2024)
- [PID-Guided Partial Alignment for Multimodal Decentralized Federated Learning](https://mlsanity.com/paper/2601.10012/) (2026)

## Abstract (arXiv)

Automatic emotion recognition is an active research topic with wide range of applications. Due to the high manual annotation cost and inevitable label ambiguity, the development of emotion recognition dataset is limited in both scale and quality. Therefore, one of the key challenges is how to build effective models with limited data resource. Previous works have explored different approaches to tackle this challenge including data enhancement, transfer learning, and semi-supervised learning etc. However, the weakness of these existing approaches includes such as training instability, large performance loss during transfer, or marginal improvement. In this work, we propose a novel semi-supervised multi-modal emotion recognition model based on cross-modality distribution matching, which leverages abundant unlabeled data to enhance the model training under the assumption that the inner emotional status is consistent at the utterance level across modalities. We conduct extensive experiments to evaluate the proposed model on two benchmark datasets, IEMOCAP and MELD. The experiment results prove that the proposed semi-supervised learning model can effectively utilize unlabeled data and combine multi-modalities to boost the emotion recognition performance, which outperforms other state-of-the-art approaches under the same condition. The proposed model also achieves competitive capacity compared with existing approaches which take advantage of additional auxiliary information such as speaker and interaction context.