# Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data

arXiv: https://arxiv.org/abs/1808.02096 · HTML: https://mlsanity.com/paper/1808.02096/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 29 of 29)

- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities](https://mlsanity.com/paper/2204.13707/) (2022)
- [Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2407.05374/) (2024)
- [Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities](https://mlsanity.com/paper/2210.15359/) (2022)
- [Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning](https://mlsanity.com/paper/2411.02793/) (2024)
- [RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates](https://mlsanity.com/paper/2312.10386/) (2023)
- [Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation](https://mlsanity.com/paper/2302.04308/) (2023)
- [Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection](https://mlsanity.com/paper/2306.02137/) (2023)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [From the Lab to the Wild: Affect Modeling via Privileged Information](https://mlsanity.com/paper/2305.10919/) (2023)
- [Maximum Likelihood Estimation for Multimodal Learning with Missing Modality](https://mlsanity.com/paper/2108.10513/) (2021)
- [HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities](https://mlsanity.com/paper/2105.07889/) (2021)
- [M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentation](https://mlsanity.com/paper/2205.02524/) (2022)
- [Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection](https://mlsanity.com/paper/2402.05294/) (2024)
- [Efficient Prompting for Continual Adaptation to Missing Modalities](https://mlsanity.com/paper/2503.00528/) (2025)
- [MVCNet: Multiview Contrastive Network for Unsupervised Representation Learning for 3D CT Lesions](https://mlsanity.com/paper/2108.07662/) (2021)
- [Discriminative Multimodal Learning via Conditional Priors in Generative Models](https://mlsanity.com/paper/2110.04616/) (2021)
- [Multi-view Multi-label Fine-grained Emotion Decoding from Human Brain Activity](https://mlsanity.com/paper/2211.02629/) (2022)
- [Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework](https://mlsanity.com/paper/2407.09029/) (2024)
- [Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate](https://mlsanity.com/paper/2505.19525/) (2025)
- [CNN Autoencoders for Hierarchical Feature Extraction and Fusion in Multi-sensor Human Activity Recognition](https://mlsanity.com/paper/2502.04489/) (2025)
- [Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities](https://mlsanity.com/paper/2508.06800/) (2025)
- [Robust Multimodal Fusion for Human Activity Recognition](https://mlsanity.com/paper/2303.04636/) (2023)
- [MACO: A Modality Adversarial and Contrastive Framework for Modality-missing Multi-modal Knowledge Graph Completion](https://mlsanity.com/paper/2308.06696/) (2023)
- [Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios](https://mlsanity.com/paper/2311.16114/) (2023)
- [Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition](https://mlsanity.com/paper/2312.15848/) (2023)
- [ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion](https://mlsanity.com/paper/2507.05624/) (2025)
- [SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts](https://mlsanity.com/paper/2610.01418/) (2026)

## Abstract (arXiv)

There are threefold challenges in emotion recognition. First, it is difficult to recognize human's emotional states only considering a single modality. Second, it is expensive to manually annotate the emotional data. Third, emotional data often suffers from missing modalities due to unforeseeable sensor malfunction or configuration issues. In this paper, we address all these problems under a novel multi-view deep generative framework. Specifically, we propose to model the statistical relationships of multi-modality emotional data using multiple modality-specific generative networks with a shared latent space. By imposing a Gaussian mixture assumption on the posterior approximation of the shared latent variables, our framework can learn the joint deep representation from multiple modalities and evaluate the importance of each modality simultaneously. To solve the labeled-data-scarcity problem, we extend our multi-view model to semi-supervised learning scenario by casting the semi-supervised classification problem as a specialized missing data imputation task. To address the missing-modality problem, we further extend our semi-supervised multi-view model to deal with incomplete data, where a missing view is treated as a latent variable and integrated out during inference. This way, the proposed overall framework can utilize all available (both labeled and unlabeled, as well as both complete and incomplete) data to improve its generalization ability. The experiments conducted on two real multi-modal emotion datasets demonstrated the superiority of our framework.