# Multimodal Sentiment Analysis To Explore the Structure of Emotions

arXiv: https://arxiv.org/abs/1805.10205 · HTML: https://mlsanity.com/paper/1805.10205/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 19 of 19)

- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies](https://mlsanity.com/paper/2108.10152/) (2021)
- [Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memes](https://mlsanity.com/paper/2303.09892/) (2023)
- [Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification](https://mlsanity.com/paper/2108.03886/) (2021)
- [UVCE-IIITT@DravidianLangTech-EACL2021: Tamil Troll Meme Classification: You need to Pay more Attention](https://mlsanity.com/paper/2104.09081/) (2021)
- [PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models](https://mlsanity.com/paper/2312.06093/) (2023)
- [Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023](https://mlsanity.com/paper/2309.07925/) (2023)
- [Problem Learning: Towards the Free Will of Machines](https://mlsanity.com/paper/2109.00177/) (2021)
- [Interpretable Multimodal Emotion Recognition using Hybrid Fusion of Speech and Image Data](https://mlsanity.com/paper/2208.11868/) (2022)
- [Overview of Memotion 3: Sentiment and Emotion Analysis of Codemixed Hinglish Memes](https://mlsanity.com/paper/2309.06517/) (2023)
- [Affective Feedback Synthesis Towards Multimodal Text and Image Data](https://mlsanity.com/paper/2203.12692/) (2022)
- [InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis](https://mlsanity.com/paper/2212.10030/) (2022)
- [NYCU-TWO at Memotion 3: Good Foundation, Good Teacher, then you have Good Meme Analysis](https://mlsanity.com/paper/2302.06078/) (2023)
- [Context-Aware Sentiment Forecasting via LLM-based Multi-Perspective Role-Playing Agents](https://mlsanity.com/paper/2505.24331/) (2025)
- [VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media](https://mlsanity.com/paper/2208.09021/) (2022)
- [EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis](https://mlsanity.com/paper/2212.08661/) (2022)
- [Beyond Words: Enhancing Desire, Emotion, and Sentiment Recognition with Non-Verbal Cues](https://mlsanity.com/paper/2509.15540/) (2025)
- [Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination](https://mlsanity.com/paper/2608.20019/) (2026)

## Abstract (arXiv)

We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual analysis and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the latent emotional state of the user. Thus, we focus on predicting the emotion word tags attached by users to their Tumblr posts, treating these as "self-reported emotions." We demonstrate that our multimodal model combining both text and image features outperforms separate models based solely on either images or text. Our model's results are interpretable, automatically yielding sensible word lists associated with emotions. We explore the structure of emotions implied by our model and compare it to what has been posited in the psychology literature, and validate our model on a set of images that have been used in psychology studies. Finally, our work also provides a useful tool for the growing academic study of images - both photographs and memes - on social networks.