# Cooperative Sentiment Agents for Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/2404.12642 · HTML: https://mlsanity.com/paper/2404.12642/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis](https://mlsanity.com/paper/1911.05544/) (2019)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning](https://mlsanity.com/paper/1802.00924/) (2018)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [Quantum-inspired Multimodal Fusion for Video Sentiment Analysis](https://mlsanity.com/paper/2103.10572/) (2021)
- [An AutoML-based Approach to Multimodal Image Sentiment Analysis](https://mlsanity.com/paper/2102.08092/) (2021)
- [False Correlation Reduction for Offline Reinforcement Learning](https://mlsanity.com/paper/2110.12468/) (2021)
- [Modality-based Factorization for Multimodal Fusion](https://mlsanity.com/paper/1811.12624/) (2018)
- [Relational Temporal Graph Reasoning for Dual-task Dialogue Language Understanding](https://mlsanity.com/paper/2306.09114/) (2023)
- [Discriminative Multiple Canonical Correlation Analysis for Information Fusion](https://mlsanity.com/paper/2103.00361/) (2021)

## Built on by (most cited first, 1 of 1)

- [SentiMM: A Multimodal Multi-Agent Framework for Sentiment Analysis in Social Media](https://mlsanity.com/paper/2508.18108/) (2025)

## Abstract (arXiv)

In this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment Agents, named Co-SA. Co-SA comprises two critical components: the Sentiment Agents Establishment (SAE) phase and the Sentiment Agents Cooperation (SAC) phase. During the SAE phase, each sentiment agent deals with an unimodal signal and highlights explicit dynamic sentiment variations within the modality via the Modality-Sentiment Disentanglement (MSD) and Deep Phase Space Reconstruction (DPSR) modules. Subsequently, in the SAC phase, Co-SA meticulously designs task-specific interaction mechanisms for sentiment agents so that coordinating multimodal signals to learn the joint representation. Specifically, Co-SA equips an independent policy model for each sentiment agent that captures significant properties within the modality. These policies are optimized mutually through the unified reward adaptive to downstream tasks. Benefitting from the rewarding mechanism, Co-SA transcends the limitation of pre-defined fusion modes and adaptively captures unimodal properties for MRL in the multimodal interaction setting. To demonstrate the effectiveness of Co-SA, we apply it to address Multimodal Sentiment Analysis (MSA) and Multimodal Emotion Recognition (MER) tasks. Our comprehensive experimental results demonstrate that Co-SA excels at discovering diverse cross-modal features, encompassing both common and complementary aspects. The code can be available at https://github.com/smwanghhh/Co-SA.