# EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes

arXiv: https://arxiv.org/abs/2307.07961 · HTML: https://mlsanity.com/paper/2307.07961/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 69 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark](https://mlsanity.com/paper/1605.02677/) (2016)
- [Affective Image Content Analysis: Two Decades Review and New Perspectives](https://mlsanity.com/paper/2106.16125/) (2021)
- [Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks](https://mlsanity.com/paper/1509.06041/) (2015)
- [Learning Multi-level Deep Representations for Image Emotion Classification](https://mlsanity.com/paper/1611.07145/) (2016)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Stimuli-Aware Visual Emotion Analysis](https://mlsanity.com/paper/2109.01812/) (2021)
- [Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning](https://mlsanity.com/paper/1804.05788/) (2018)
- [AngryBERT: Joint Learning Target and Emotion for Hate Speech Detection](https://mlsanity.com/paper/2103.11800/) (2021)
- [SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network](https://mlsanity.com/paper/2110.12334/) (2021)
- [PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression](https://mlsanity.com/paper/1909.05693/) (2019)
- [Recognition of Advertisement Emotions with Application to Computational Advertising](https://mlsanity.com/paper/1904.01778/) (2019)

## Built on by (most cited first, 30 of 82)

- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning](https://mlsanity.com/paper/2406.11161/) (2024)
- [EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning](https://mlsanity.com/paper/2404.16670/) (2024)
- [GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition](https://mlsanity.com/paper/2312.04293/) (2023)
- [EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2401.04608/) (2024)
- [EmoEdit: Evoking Emotions through Image Manipulation](https://mlsanity.com/paper/2405.12661/) (2024)
- [Evaluating Vision-Language Models for Emotion Recognition](https://mlsanity.com/paper/2502.05660/) (2025)
- [UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries](https://mlsanity.com/paper/2507.23372/) (2025)
- [Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition](https://mlsanity.com/paper/2505.19694/) (2025)
- [Make Me Happier: Evoking Emotions Through Image Diffusion Models](https://mlsanity.com/paper/2403.08255/) (2024)
- [EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment](https://mlsanity.com/paper/2504.16405/) (2025)
- [Low-Resource Vision Challenges for Foundation Models](https://mlsanity.com/paper/2401.04716/) (2024)
- [Affective Image Editing: Shaping Emotional Factors via Text Descriptions](https://mlsanity.com/paper/2505.18699/) (2025)
- [SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information](https://mlsanity.com/paper/2409.14083/) (2024)
- [EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generation](https://mlsanity.com/paper/2506.03652/) (2025)
- [MAPO: Mixed Advantage Policy Optimization](https://mlsanity.com/paper/2509.18849/) (2025)
- [CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation](https://mlsanity.com/paper/2508.03535/) (2025)
- [StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2409.00304/) (2024)
- [AI-generated Image Quality Assessment in Visual Communication](https://mlsanity.com/paper/2412.15677/) (2024)
- [EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis](https://mlsanity.com/paper/2511.12554/) (2025)
- [Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks](https://mlsanity.com/paper/2607.07907/) (2026)
- [Socratis: Are large multimodal models emotionally aware?](https://mlsanity.com/paper/2308.16741/) (2023)
- [EmoFeedback2: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedback](https://mlsanity.com/paper/2511.19982/) (2025)
- [Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding](https://mlsanity.com/paper/2601.16449/) (2026)
- [VIBE: Can a VLM Read the Room?](https://mlsanity.com/paper/2506.11162/) (2025)
- [MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition](https://mlsanity.com/paper/2507.18929/) (2025)
- [Instant Preference Alignment for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.17718/) (2025)
- [EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration](https://mlsanity.com/paper/2512.15528/) (2025)
- [EmoCtrl: Controllable Emotional Image Content Generation](https://mlsanity.com/paper/2512.22437/) (2025)
- [EmoKGEdit: Training-free Affective Injection via Visual Cue Transformation](https://mlsanity.com/paper/2601.12326/) (2026)

## Abstract (arXiv)

Visual Emotion Analysis (VEA) aims at predicting people's emotional responses to visual stimuli. This is a promising, yet challenging, task in affective computing, which has drawn increasing attention in recent years. Most of the existing work in this area focuses on feature design, while little attention has been paid to dataset construction. In this work, we introduce EmoSet, the first large-scale visual emotion dataset annotated with rich attributes, which is superior to existing datasets in four aspects: scale, annotation richness, diversity, and data balance. EmoSet comprises 3.3 million images in total, with 118,102 of these images carefully labeled by human annotators, making it five times larger than the largest existing dataset. EmoSet includes images from social networks, as well as artistic images, and it is well balanced between different emotion categories. Motivated by psychological studies, in addition to emotion category, each image is also annotated with a set of describable emotion attributes: brightness, colorfulness, scene type, object class, facial expression, and human action, which can help understand visual emotions in a precise and interpretable way. The relevance of these emotion attributes is validated by analyzing the correlations between them and visual emotion, as well as by designing an attribute module to help visual emotion recognition. We believe EmoSet will bring some key insights and encourage further research in visual emotion analysis and understanding. Project page: https://vcc.tech/EmoSet.