# Holistic Visual-Textual Sentiment Analysis with Prior Models

arXiv: https://arxiv.org/abs/2211.12981 · HTML: https://mlsanity.com/paper/2211.12981/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks](https://mlsanity.com/paper/1509.06041/) (2015)
- [CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection](https://mlsanity.com/paper/2204.05515/) (2022)
- [When Saliency Meets Sentiment: Understanding How Image Content Invokes Emotion and Sentiment](https://mlsanity.com/paper/1611.04636/) (2016)
- [A Fair and Comprehensive Comparison of Multimodal Tweet Sentiment Analysis Methods](https://mlsanity.com/paper/2106.08829/) (2021)

## Built on by (most cited first, 3 of 3)

- [GPT-4V(ision) as A Social Media Analysis Engine](https://mlsanity.com/paper/2311.07547/) (2023)
- [Robust Multimodal Sentiment Analysis via Double Information Bottleneck](https://mlsanity.com/paper/2511.01444/) (2025)
- [An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2510.23617/) (2025)

## Abstract (arXiv)

Visual-textual sentiment analysis aims to predict sentiment with the input of a pair of image and text, which poses a challenge in learning effective features for diverse input images. To address this, we propose a holistic method that achieves robust visual-textual sentiment analysis by exploiting a rich set of powerful pre-trained visual and textual prior models. The proposed method consists of four parts: (1) a visual-textual branch to learn features directly from data for sentiment analysis, (2) a visual expert branch with a set of pre-trained "expert" encoders to extract selected semantic visual features, (3) a CLIP branch to implicitly model visual-textual correspondence, and (4) a multimodal feature fusion network based on BERT to fuse multimodal features and make sentiment predictions. Extensive experiments on three datasets show that our method produces better visual-textual sentiment analysis performance than existing methods.