# Target-Dependent Multimodal Sentiment Analysis Via Employing Visual-to Emotional-Caption Translation Network using Visual-Caption Pairs

arXiv: https://arxiv.org/abs/2408.10248 · HTML: https://mlsanity.com/paper/2408.10248/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [Knowing What, How and Why: A Near Complete Solution for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1911.01616/) (2019)
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2203.01054/) (2022)
- [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1910.00883/) (2019)
- [Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation](https://mlsanity.com/paper/2108.01682/) (2021)
- [Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition](https://mlsanity.com/paper/2009.00210/) (2020)
- [Aspect-based Sentiment Analysis of Scientific Reviews](https://mlsanity.com/paper/2006.03257/) (2020)
- [A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1609.02745/) (2016)
- [Facial Emotion Recognition: State of the Art Performance on FER2013](https://mlsanity.com/paper/2105.03588/) (2021)
- [A Survey on Aspect-Based Sentiment Classification](https://mlsanity.com/paper/2203.14266/) (2022)
- [Targeted aspect based multimodal sentiment analysis:an attention capsule extraction and multi-head fusion network](https://mlsanity.com/paper/2103.07659/) (2021)

## Abstract (arXiv)

The natural language processing and multimedia field has seen a notable surge in interest in multimodal sentiment recognition. Hence, this study aims to employ Target-Dependent Multimodal Sentiment Analysis (TDMSA) to identify the level of sentiment associated with every target (aspect) stated within a multimodal post consisting of a visual-caption pair. Despite the recent advancements in multimodal sentiment recognition, there has been a lack of explicit incorporation of emotional clues from the visual modality, specifically those pertaining to facial expressions. The challenge at hand is to proficiently obtain visual and emotional clues and subsequently synchronise them with the textual content. In light of this fact, this study presents a novel approach called the Visual-to-Emotional-Caption Translation Network (VECTN) technique. The primary objective of this strategy is to effectively acquire visual sentiment clues by analysing facial expressions. Additionally, it effectively aligns and blends the obtained emotional clues with the target attribute of the caption mode. The experimental findings demonstrate that our methodology is capable of producing ground-breaking outcomes when applied to two publicly accessible multimodal Twitter datasets, namely, Twitter-2015 and Twitter-2017. The experimental results show that the suggested model achieves an accuracy of 81.23% and a macro-F1 of 80.61% on the Twitter-15 dataset, while 77.42% and 75.19% on the Twitter-17 dataset, respectively. The observed improvement in performance reveals that our model is better than others when it comes to collecting target-level sentiment in multimodal data using the expressions of the face.