# Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/1609.05244 · HTML: https://mlsanity.com/paper/1609.05244/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2016.

## Built on by (most cited first, 29 of 29)

- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](https://mlsanity.com/paper/2107.13669/) (2021)
- [Nonlinear Invariant Risk Minimization: A Causal Approach](https://mlsanity.com/paper/2102.12353/) (2021)
- [Multimodal End-to-End Sparse Model for Emotion Recognition](https://mlsanity.com/paper/2103.09666/) (2021)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning](https://mlsanity.com/paper/2312.05735/) (2023)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [The Two Dimensions of Worst-case Training and the Integrated Effect for Out-of-domain Generalization](https://mlsanity.com/paper/2204.04384/) (2022)
- [Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2305.07611/) (2023)
- [Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data](https://mlsanity.com/paper/2106.13213/) (2021)
- [Hybrid Multimodal Fusion for Humor Detection](https://mlsanity.com/paper/2209.11949/) (2022)
- [From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation](https://mlsanity.com/paper/2110.08270/) (2021)
- [LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences](https://mlsanity.com/paper/2112.01697/) (2021)
- [TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.06536/) (2025)
- [Mitigating Confounding in Speech-Based Dementia Detection through Weight Masking](https://mlsanity.com/paper/2506.05610/) (2025)
- [Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalance](https://mlsanity.com/paper/2605.06289/) (2026)
- [CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations](https://mlsanity.com/paper/2209.07424/) (2022)
- [FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference](https://mlsanity.com/paper/2209.10170/) (2022)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Variational Learning of Disentangled Representations](https://mlsanity.com/paper/2506.17182/) (2025)
- [Will Multi-modal Data Improves Few-shot Learning?](https://mlsanity.com/paper/2107.11853/) (2021)
- [Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos](https://mlsanity.com/paper/2206.07981/) (2022)
- [Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization](https://mlsanity.com/paper/2311.15145/) (2023)
- ["Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?](https://mlsanity.com/paper/2501.04138/) (2025)
- [An Attention-based Model for Robust Forecasting with Missing Modality](https://mlsanity.com/paper/2606.13970/) (2026)

## Abstract (arXiv)

Multimodal sentiment analysis is drawing an increasing amount of attention these days. It enables mining of opinions in video reviews which are now available aplenty on online platforms. However, multimodal sentiment analysis has only a few high-quality data sets annotated for training machine learning algorithms. These limited resources restrict the generalizability of models, where, for example, the unique characteristics of a few speakers (e.g., wearing glasses) may become a confounding factor for the sentiment classification task. In this paper, we propose a Select-Additive Learning (SAL) procedure that improves the generalizability of trained neural networks for multimodal sentiment analysis. In our experiments, we show that our SAL approach improves prediction accuracy significantly in all three modalities (verbal, acoustic, visual), as well as in their fusion. Our results show that SAL, even when trained on one dataset, achieves good generalization across two new test datasets.