# CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

arXiv: https://arxiv.org/abs/2204.05515 · HTML: https://mlsanity.com/paper/2204.05515/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://mlsanity.com/paper/1804.09541/) (2018)
- [Deep Learning for Sentiment Analysis : A Survey](https://mlsanity.com/paper/1801.07883/) (2018)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Text Level Graph Neural Network for Text Classification](https://mlsanity.com/paper/1910.02356/) (2019)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [Gated Mechanism for Attention Based Multimodal Sentiment Analysis](https://mlsanity.com/paper/2003.01043/) (2020)
- [Are Safer Looking Neighborhoods More Lively? A Multimodal Investigation into Urban Life](https://mlsanity.com/paper/1608.00462/) (2016)

## Built on by (most cited first, 21 of 21)

- [MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks](https://mlsanity.com/paper/2310.09036/) (2023)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.11009/) (2023)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts](https://mlsanity.com/paper/2211.06607/) (2022)
- [Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing](https://mlsanity.com/paper/2509.15361/) (2025)
- [Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decomposition](https://mlsanity.com/paper/2407.07026/) (2024)
- [End-to-end Semantic-centric Video-based Multimodal Affective Computing](https://mlsanity.com/paper/2408.07694/) (2024)
- [An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capability](https://mlsanity.com/paper/2505.16193/) (2025)
- [Holistic Visual-Textual Sentiment Analysis with Prior Models](https://mlsanity.com/paper/2211.12981/) (2022)
- [What Matters to Enhance Traffic Rule Compliance of Imitation Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2309.07808/) (2023)
- [Memory-Inspired Temporal Prompt Interaction for Text-Image Classification](https://mlsanity.com/paper/2401.14856/) (2024)
- [Robust Multimodal Sentiment Analysis via Double Information Bottleneck](https://mlsanity.com/paper/2511.01444/) (2025)
- [Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion](https://mlsanity.com/paper/2511.18751/) (2025)
- [Exploring Multimodal Sentiment Analysis via CBAM Attention and Double-layer BiLSTM Architecture](https://mlsanity.com/paper/2303.14708/) (2023)
- [Data Uncertainty-Aware Learning for Multimodal Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2412.01249/) (2024)
- [Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localization](https://mlsanity.com/paper/2509.13474/) (2025)
- [Beyond Words: Enhancing Desire, Emotion, and Sentiment Recognition with Non-Verbal Cues](https://mlsanity.com/paper/2509.15540/) (2025)
- [An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2510.23617/) (2025)
- [Improve Contrastive Clustering Performance by Multiple Fusing-Augmenting ViT Blocks](https://mlsanity.com/paper/2511.08883/) (2025)

## Abstract (arXiv)

Compared with unimodal data, multimodal data can provide more features to help the model analyze the sentiment of data. Previous research works rarely consider token-level feature fusion, and few works explore learning the common features related to sentiment in multimodal data to help the model fuse multimodal features. In this paper, we propose a Contrastive Learning and Multi-Layer Fusion (CLMLF) method for multimodal sentiment detection. Specifically, we first encode text and image to obtain hidden representations, and then use a multi-layer fusion module to align and fuse the token-level features of text and image. In addition to the sentiment analysis task, we also designed two contrastive learning tasks, label based contrastive learning and data based contrastive learning tasks, which will help the model learn common features related to sentiment in multimodal data. Extensive experiments conducted on three publicly available multimodal datasets demonstrate the effectiveness of our approach for multimodal sentiment detection compared with existing methods. The codes are available for use at https://github.com/Link-Li/CLMLF