# Transfer Learning with Joint Fine-Tuning for Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/2210.05790 · HTML: https://mlsanity.com/paper/2210.05790/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Deep Learning for Sentiment Analysis : A Survey](https://mlsanity.com/paper/1801.07883/) (2018)
- [Sentiment Analysis Based on Deep Learning: A Comparative Study](https://mlsanity.com/paper/2006.03541/) (2020)
- [Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction](https://mlsanity.com/paper/1508.05056/) (2015)

## Built on by (most cited first, 1 of 1)

- [Multimodal Sentiment Analysis Based on Causal Reasoning](https://mlsanity.com/paper/2412.07292/) (2024)

## Abstract (arXiv)

Most existing methods focus on sentiment analysis of textual data. However, recently there has been a massive use of images and videos on social platforms, motivating sentiment analysis from other modalities. Current studies show that exploring other modalities (e.g., images) increases sentiment analysis performance. State-of-the-art multimodal models, such as CLIP and VisualBERT, are pre-trained on datasets with the text paired with images. Although the results obtained by these models are promising, pre-training and sentiment analysis fine-tuning tasks of these models are computationally expensive. This paper introduces a transfer learning approach using joint fine-tuning for sentiment analysis. Our proposal achieved competitive results using a more straightforward alternative fine-tuning strategy that leverages different pre-trained unimodal models and efficiently combines them in a multimodal space. Moreover, our proposal allows flexibility when incorporating any pre-trained model for texts and images during the joint fine-tuning stage, being especially interesting for sentiment classification in low-resource scenarios.