mlsanityData to 7 Oct 2026

CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

arXiv 2204.05515 ↗First posted 12 Apr 2022cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Compared with unimodal data, multimodal data can provide more features to help the model analyze the sentiment of data. Previous research works rarely consider token-level feature fusion, and few works explore learning the common features related to sentiment in multimodal data to help the model fuse multimodal features. In this paper, we propose a Contrastive Learning and Multi-Layer Fusion (CLMLF) method for multimodal sentiment detection. Specifically, we first encode text and image to obtain hidden representations, and then use a multi-layer fusion module to align and fuse the token-level features of text and image. In addition to the sentiment analysis task, we also designed two contrastive learning tasks, label based contrastive learning and data based contrastive learning tasks, which will help the model learn common features related to sentiment in multimodal data. Extensive experiments conducted on three publicly available multimodal datasets demonstrate the effectiveness of our approach for multimodal sentiment detection compared with existing methods. The codes are available for use at https://github.com/Link-Li/CLMLF

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

27 of 52 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  12. 2021Transformers in Vision: A Surveycited by 1,009
15 more
  1. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  2. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  3. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  4. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  5. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  6. 2021A Survey of Transformerscited by 399
  7. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  8. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  9. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  10. 2018Deep Learning for Sentiment Analysis : A Surveycited by 139
  11. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  12. 2019Text Level Graph Neural Network for Text Classificationcited by 60
  13. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  14. 2020Gated Mechanism for Attention Based Multimodal Sentiment Analysiscited by 17
  15. 2016Are Safer Looking Neighborhoods More Lively? A Multimodal Investigation into Urban Lifecited by 9

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 6; 2025: 9.

21 papers.

Sort:

By area:

  1. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Taskscited by 18
  2. 2024Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Surveycited by 16
  3. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  4. 2023Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognitioncited by 14
  5. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  6. 2022Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Promptscited by 9
  7. 2025Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routingcited by 5
  8. 2024Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decompositioncited by 4
  9. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  10. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capabilitycited by 4
  11. 2022Holistic Visual-Textual Sentiment Analysis with Prior Modelscited by 3
  12. 2023What Matters to Enhance Traffic Rule Compliance of Imitation Learning for End-to-End Autonomous Drivingcited by 2
9 more
  1. 2024Memory-Inspired Temporal Prompt Interaction for Text-Image Classificationcited by 1
  2. 2025Robust Multimodal Sentiment Analysis via Double Information Bottleneckcited by 1
  3. 2025Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusioncited by 1
  4. 2023Exploring Multimodal Sentiment Analysis via CBAM Attention and Double-layer BiLSTM Architecturecited by 0
  5. 2024Data Uncertainty-Aware Learning for Multimodal Aspect-based Sentiment Analysiscited by 0
  6. 2025Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localizationcited by 0
  7. 2025Beyond Words: Enhancing Desire, Emotion, and Sentiment Recognition with Non-Verbal Cuescited by 0
  8. 2025An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysiscited by 0
  9. 2025Improve Contrastive Clustering Performance by Multiple Fusing-Augmenting ViT Blockscited by 0