# Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/2210.15824 · HTML: https://mlsanity.com/paper/2210.15824/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Supporting Clustering with Contrastive Learning](https://mlsanity.com/paper/2103.12953/) (2021)
- [M-SENA: An Integrated Platform for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2203.12441/) (2022)
- [Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2111.05222/) (2021)
- [Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis](https://mlsanity.com/paper/2207.11652/) (2022)

## Built on by (most cited first, 2 of 2)

- [Hybrid Multi-stage Decoding for Few-shot NER with Entity-aware Contrastive Learning](https://mlsanity.com/paper/2404.06970/) (2024)
- [MIAR: Modality Interaction and Alignment Representation Fuison for Multimodal Emotion](https://mlsanity.com/paper/2601.02414/) (2026)

## Abstract (arXiv)

Modality representation learning is an important problem for multimodal sentiment analysis (MSA), since the highly distinguishable representations can contribute to improving the analysis effect. Previous works of MSA have usually focused on multimodal fusion strategies, and the deep study of modal representation learning was given less attention. Recently, contrastive learning has been confirmed effective at endowing the learned representation with stronger discriminate ability. Inspired by this, we explore the improvement approaches of modality representation with contrastive learning in this study. To this end, we devise a three-stages framework with multi-view contrastive learning to refine representations for the specific objectives. At the first stage, for the improvement of unimodal representations, we employ the supervised contrastive learning to pull samples within the same class together while the other samples are pushed apart. At the second stage, a self-supervised contrastive learning is designed for the improvement of the distilled unimodal representations after cross-modal interaction. At last, we leverage again the supervised contrastive learning to enhance the fused multimodal representation. After all the contrast trainings, we next achieve the classification task based on frozen representations. We conduct experiments on three open datasets, and results show the advance of our model.