# Multi-Modality Collaborative Learning for Sentiment Analysis

arXiv: https://arxiv.org/abs/2501.12424 · HTML: https://mlsanity.com/paper/2501.12424/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 64 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/1905.05812/) (2019)
- [Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation](https://mlsanity.com/paper/2002.08267/) (2020)
- [Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Prediction](https://mlsanity.com/paper/1805.00705/) (2018)
- [Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations](https://mlsanity.com/paper/2407.04955/) (2024)
- [An AutoML-based Approach to Multimodal Image Sentiment Analysis](https://mlsanity.com/paper/2102.08092/) (2021)
- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects](https://mlsanity.com/paper/2106.14174/) (2021)
- [Video-based Cross-modal Auxiliary Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.13954/) (2022)
- [On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2210.15937/) (2022)

## Built on by (most cited first, 1 of 1)

- [MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Models](https://mlsanity.com/paper/2507.05591/) (2025)

## Abstract (arXiv)

Multimodal sentiment analysis (MSA) identifies individuals' sentiment states in videos by integrating visual, audio, and text modalities. Despite progress in existing methods, the inherent modality heterogeneity limits the effective capture of interactive sentiment features across modalities. In this paper, by introducing a Multi-Modality Collaborative Learning (MMCL) framework, we facilitate cross-modal interactions and capture enhanced and complementary features from modality-common and modality-specific representations, respectively. Specifically, we design a parameter-free decoupling module and separate uni-modality into modality-common and modality-specific components through semantics assessment of cross-modal elements. For modality-specific representations, inspired by the act-reward mechanism in reinforcement learning, we design policy models to adaptively mine complementary sentiment features under the guidance of a joint reward. For modality-common representations, intra-modal attention is employed to highlight crucial components, playing enhanced roles among modalities. Experimental results, including superiority evaluations on four databases, effectiveness verification of each module, and assessment of complementary features, demonstrate that MMCL successfully learns collaborative features across modalities and significantly improves performance. The code can be available at https://github.com/smwanghhh/MMCL.