# Stimuli-Aware Visual Emotion Analysis

arXiv: https://arxiv.org/abs/2109.01812 · HTML: https://mlsanity.com/paper/2109.01812/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 79 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark](https://mlsanity.com/paper/1605.02677/) (2016)
- [Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks](https://mlsanity.com/paper/1509.06041/) (2015)
- [Learning Multi-level Deep Representations for Image Emotion Classification](https://mlsanity.com/paper/1611.07145/) (2016)
- [DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1410.8586/) (2014)
- [Linear Disentangled Representation Learning for Facial Actions](https://mlsanity.com/paper/1701.03102/) (2017)
- [Hierarchical Sparse and Collaborative Low-Rank Representation for Emotion Recognition](https://mlsanity.com/paper/1410.1606/) (2014)

## Built on by (most cited first, 26 of 26)

- [EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes](https://mlsanity.com/paper/2307.07961/) (2023)
- [EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning](https://mlsanity.com/paper/2404.16670/) (2024)
- [EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2401.04608/) (2024)
- [EmoEdit: Evoking Emotions through Image Manipulation](https://mlsanity.com/paper/2405.12661/) (2024)
- [Evaluating Vision-Language Models for Emotion Recognition](https://mlsanity.com/paper/2502.05660/) (2025)
- [SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network](https://mlsanity.com/paper/2110.12334/) (2021)
- [UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries](https://mlsanity.com/paper/2507.23372/) (2025)
- [Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition](https://mlsanity.com/paper/2505.19694/) (2025)
- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [Affective Image Editing: Shaping Emotional Factors via Text Descriptions](https://mlsanity.com/paper/2505.18699/) (2025)
- [CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation](https://mlsanity.com/paper/2508.03535/) (2025)
- [Color-Emotion Associations in Art: Fuzzy Approach](https://mlsanity.com/paper/2311.18518/) (2023)
- [StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2409.00304/) (2024)
- [Seeking Subjectivity in Visual Emotion Distribution Learning](https://mlsanity.com/paper/2207.11875/) (2022)
- [Training A Small Emotional Vision Language Model for Visual Art Comprehension](https://mlsanity.com/paper/2403.11150/) (2024)
- [EmoCtrl: Controllable Emotional Image Content Generation](https://mlsanity.com/paper/2512.22437/) (2025)
- [UniEmoX: Cross-modal Semantic-Guided Large-Scale Pretraining for Universal Scene Emotion Perception](https://mlsanity.com/paper/2409.18877/) (2024)
- [Teaching Human Behavior Improves Content Understanding Abilities Of LLMs](https://mlsanity.com/paper/2405.00942/) (2024)
- [Don't Judge Before You CLIP: A Unified Approach for Perceptual Tasks](https://mlsanity.com/paper/2503.13260/) (2025)
- [Multi-channel Emotion Analysis for Consensus Reaching in Group Movie Recommendation Systems](https://mlsanity.com/paper/2404.13778/) (2024)
- [Omni-SILA: Towards Omni-scene Driven Visual Sentiment Identifying, Locating and Attributing in Videos](https://mlsanity.com/paper/2503.00049/) (2025)
- [EmoSEM: Segment and Explain Emotion Stimuli in Visual Art](https://mlsanity.com/paper/2504.14658/) (2025)
- [Read-Best Is Not Steer-Best: A Probing--Steering Layer Dissociation in Omni-Modal Large Language Models](https://mlsanity.com/paper/2609.22135/) (2026)
- [Attributes-aware Visual Emotion Representation Learning](https://mlsanity.com/paper/2504.06578/) (2025)
- [ProFocus: Interpreting Affective Experience in Artistic Images with Progressive Visual Focusing](https://mlsanity.com/paper/2608.13974/) (2026)
- [Balancing Emotional Alignment and Semantic Consistency in Image Generation via Reinforcement Learning with Valence-Arousal Anchoring](https://mlsanity.com/paper/2609.12830/) (2026)

## Abstract (arXiv)

Visual emotion analysis (VEA) has attracted great attention recently, due to the increasing tendency of expressing and understanding emotions through images on social networks. Different from traditional vision tasks, VEA is inherently more challenging since it involves a much higher level of complexity and ambiguity in human cognitive process. Most of the existing methods adopt deep learning techniques to extract general features from the whole image, disregarding the specific features evoked by various emotional stimuli. Inspired by the textitStimuli-Organism-Response (S-O-R) emotion model in psychological theory, we proposed a stimuli-aware VEA method consisting of three stages, namely stimuli selection (S), feature extraction (O) and emotion prediction (R). First, specific emotional stimuli (i.e., color, object, face) are selected from images by employing the off-the-shelf tools. To the best of our knowledge, it is the first time to introduce stimuli selection process into VEA in an end-to-end network. Then, we design three specific networks, i.e., Global-Net, Semantic-Net and Expression-Net, to extract distinct emotional features from different stimuli simultaneously. Finally, benefiting from the inherent structure of Mikel's wheel, we design a novel hierarchical cross-entropy loss to distinguish hard false examples from easy ones in an emotion-specific manner. Experiments demonstrate that the proposed method consistently outperforms the state-of-the-art approaches on four public visual emotion datasets. Ablation study and visualizations further prove the validity and interpretability of our method.