mlsanityData to 13 Oct 2026

WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge

arXiv 2401.06659 ↗First posted 12 Jan 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Sentiment analysis is rapidly advancing by utilizing various data modalities (e.g., text, image). However, most previous works relied on superficial information, neglecting the incorporation of contextual world knowledge (e.g., background information derived from but beyond the given image and text pairs) and thereby restricting their ability to achieve better multimodal sentiment analysis (MSA). In this paper, we proposed a plug-in framework named WisdoM, to leverage the contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced MSA. WisdoM utilizes LVLMs to comprehensively analyze both images and corresponding texts, simultaneously generating pertinent context. To reduce the noise in the context, we also introduce a training-free contextual fusion mechanism. Experiments across diverse granularities of MSA tasks consistently demonstrate that our approach has substantial improvements (brings an average +1.96% F1 score among five advanced methods) over several state-of-the-art methods.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 41 of its 77 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

41 of 77 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,360
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,204
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
29 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,205
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 933
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 898
  5. 2019Are Sixteen Heads Really Better than One?cited by 807
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 600
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 546
  8. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 317
  9. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  10. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  11. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 263
  12. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  13. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
  14. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  15. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  16. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  17. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  18. 2021CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learningcited by 69
  19. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 60
  20. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 55
  21. 2020SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Fillingcited by 42
  22. 2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
  23. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  24. 2018Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modelingcited by 39
  25. 2023Augmented Large Language Models with Parametric Knowledge Guidingcited by 37
  26. 2023Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysiscited by 29
  27. 2022Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysiscited by 21
  28. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  29. 2023AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysiscited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024DLF: Disentangled-Language-Focused Multimodal Sentiment Analysiscited by 27
  2. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 25
  3. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  4. 2024Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Surveycited by 16
  5. 2024MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysiscited by 10
  6. 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
  7. 2025MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognitioncited by 9
  8. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
  9. 2025Can Large Vision-Language Models Understand Multimodal Sarcasm?cited by 5
  10. 2024Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detectioncited by 4
  11. 2024Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuningcited by 3
  12. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
4 more
  1. 2026Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 1
  2. 2025Beyond Words: Enhancing Desire, Emotion, and Sentiment Recognition with Non-Verbal Cuescited by 0
  3. 2026Explainable Multimodal Aspect-Based Sentiment Analysis with Dependency-guided Large Language Modelcited by 0
  4. 2026Towards High-Resolution Visual Perception via Hierarchical Entity Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal LLMs See Sentiment
  2. 2026Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
  3. 2026Explicit Representation Alignment for Multimodal Sentiment Analysis
  4. 2026MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
  5. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
  6. 2026Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correction
  7. 2025MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs
  8. 2026QASA: Quality-Aware Semantic Augmentation for Robust Multimodal Sentiment Analysis