# Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias

arXiv: https://arxiv.org/abs/1808.02212 · HTML: https://mlsanity.com/paper/1808.02212/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2018.

## Built on by (most cited first, 30 of 41)

- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [A Survey on Bias in Visual Datasets](https://mlsanity.com/paper/2107.07919/) (2021)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [Affective Image Content Analysis: Two Decades Review and New Perspectives](https://mlsanity.com/paper/2106.16125/) (2021)
- [EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning](https://mlsanity.com/paper/2404.16670/) (2024)
- [Context De-confounded Emotion Recognition](https://mlsanity.com/paper/2303.11921/) (2023)
- [Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization](https://mlsanity.com/paper/2108.05449/) (2021)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Affect2MM: Affective Analysis of Multimedia Content Using Emotion Causality](https://mlsanity.com/paper/2103.06541/) (2021)
- [Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition](https://mlsanity.com/paper/2505.19694/) (2025)
- [Exploring Visual Engagement Signals for Representation Learning](https://mlsanity.com/paper/2104.07767/) (2021)
- [Towards Unbiased Visual Emotion Recognition via Causal Intervention](https://mlsanity.com/paper/2107.12096/) (2021)
- [Towards Context-Aware Emotion Recognition Debiasing from a Causal Demystification Perspective via De-confounded Training](https://mlsanity.com/paper/2407.04963/) (2024)
- [EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generation](https://mlsanity.com/paper/2506.03652/) (2025)
- [MAPO: Mixed Advantage Policy Optimization](https://mlsanity.com/paper/2509.18849/) (2025)
- [Bias Reducing Multitask Learning on Mental Health Prediction](https://mlsanity.com/paper/2208.03621/) (2022)
- [Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks](https://mlsanity.com/paper/2607.07907/) (2026)
- [Pulling Up by the Causal Bootstraps: Causal Data Augmentation for Pre-training Debiasing](https://mlsanity.com/paper/2108.12510/) (2021)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [Socratis: Are large multimodal models emotionally aware?](https://mlsanity.com/paper/2308.16741/) (2023)
- [On the use of Vision-Language models for Visual Sentiment Analysis: a study on CLIP](https://mlsanity.com/paper/2310.12062/) (2023)
- [A Review and Roadmap of Deep Learning Causal Discovery in Different Variable Paradigms](https://mlsanity.com/paper/2209.06367/) (2022)
- [AU-Aware Vision Transformers for Biased Facial Expression Recognition](https://mlsanity.com/paper/2211.06609/) (2022)
- [Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition](https://mlsanity.com/paper/2303.04896/) (2023)
- [Delving Deep into Engagement Prediction of Short Videos](https://mlsanity.com/paper/2410.00289/) (2024)
- [EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration](https://mlsanity.com/paper/2512.15528/) (2025)
- [Impressions: Understanding Visual Semiotics and Aesthetic Impact](https://mlsanity.com/paper/2310.17887/) (2023)
- [UniEmoX: Cross-modal Semantic-Guided Large-Scale Pretraining for Universal Scene Emotion Perception](https://mlsanity.com/paper/2409.18877/) (2024)
- [Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach](https://mlsanity.com/paper/2509.21950/) (2025)

## Abstract (arXiv)

While machine learning approaches to visual emotion recognition offer great promise, current methods consider training and testing models on small scale datasets covering limited visual emotion concepts. Our analysis identifies an important but long overlooked issue of existing visual emotion benchmarks in the form of dataset biases. We design a series of tests to show and measure how such dataset biases obstruct learning a generalizable emotion recognition model. Based on our analysis, we propose a webly supervised approach by leveraging a large quantity of stock image data. Our approach uses a simple yet effective curriculum guided training strategy for learning discriminative emotion features. We discover that the models learned using our large scale stock image dataset exhibit significantly better generalization ability than the existing datasets without the manual collection of even a single label. Moreover, visual representation learned using our approach holds a lot of promise across a variety of tasks on different image and video datasets.