# Towards Unbiased Visual Emotion Recognition via Causal Intervention

arXiv: https://arxiv.org/abs/2107.12096 · HTML: https://mlsanity.com/paper/2107.12096/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions](https://mlsanity.com/paper/1707.06422/) (2017)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)
- [Interventional Few-Shot Learning](https://mlsanity.com/paper/2009.13000/) (2020)
- [Learning Independent Causal Mechanisms](https://mlsanity.com/paper/1712.00961/) (2017)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [Image Counterfactual Sensitivity Analysis for Detecting Unintended Bias](https://mlsanity.com/paper/1906.06439/) (2019)
- [Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark](https://mlsanity.com/paper/1605.02677/) (2016)
- [Deconfounded Image Captioning: A Causal Retrospect](https://mlsanity.com/paper/2003.03923/) (2020)
- [Visual Causal Feature Learning](https://mlsanity.com/paper/1412.2309/) (2014)
- [Counterfactuals uncover the modular structure of deep generative models](https://mlsanity.com/paper/1812.03253/) (2018)
- [Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition](https://mlsanity.com/paper/2104.05160/) (2021)
- [Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks](https://mlsanity.com/paper/1509.06041/) (2015)
- [Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications](https://mlsanity.com/paper/1606.03237/) (2016)
- [Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias](https://mlsanity.com/paper/1808.02212/) (2018)
- [Facial Motion Prior Networks for Facial Expression Recognition](https://mlsanity.com/paper/1902.08788/) (2019)
- [Joint Deep Learning of Facial Expression Synthesis and Recognition](https://mlsanity.com/paper/2002.02194/) (2020)
- [GeoConv: Geodesic Guided Convolution for Facial Action Unit Recognition](https://mlsanity.com/paper/2003.03055/) (2020)

## Built on by (most cited first, 11 of 11)

- [Domain Generalization -- A Causal Perspective](https://mlsanity.com/paper/2209.15177/) (2022)
- [Bias Reducing Multitask Learning on Mental Health Prediction](https://mlsanity.com/paper/2208.03621/) (2022)
- [Towards Deconfounded Image-Text Matching with Causal Inference](https://mlsanity.com/paper/2408.12292/) (2024)
- [Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer](https://mlsanity.com/paper/2404.17205/) (2024)
- [A Review and Roadmap of Deep Learning Causal Discovery in Different Variable Paradigms](https://mlsanity.com/paper/2209.06367/) (2022)
- [Robust Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2502.16129/) (2025)
- [Meta-Auxiliary Learning for Micro-Expression Recognition](https://mlsanity.com/paper/2404.12024/) (2024)
- [CausalAffect: Causal Discovery for Facial Affective Understanding](https://mlsanity.com/paper/2512.00456/) (2025)
- [Dynamic Causal Disentanglement Model for Dialogue Emotion Detection](https://mlsanity.com/paper/2309.06928/) (2023)
- [CIParsing: Unifying Causality Properties into Multiple Human Parsing](https://mlsanity.com/paper/2308.12218/) (2023)
- [Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval](https://mlsanity.com/paper/2511.15201/) (2025)

## Abstract (arXiv)

Although much progress has been made in visual emotion recognition, researchers have realized that modern deep networks tend to exploit dataset characteristics to learn spurious statistical associations between the input and the target. Such dataset characteristics are usually treated as dataset bias, which damages the robustness and generalization performance of these recognition systems. In this work, we scrutinize this problem from the perspective of causal inference, where such dataset characteristic is termed as a confounder which misleads the system to learn the spurious correlation. To alleviate the negative effects brought by the dataset bias, we propose a novel Interventional Emotion Recognition Network (IERN) to achieve the backdoor adjustment, which is one fundamental deconfounding technique in causal inference. Specifically, IERN starts by disentangling the dataset-related context feature from the actual emotion feature, where the former forms the confounder. The emotion feature will then be forced to see each confounder stratum equally before being fed into the classifier. A series of designed tests validate the efficacy of IERN, and experiments on three emotion benchmarks demonstrate that IERN outperforms state-of-the-art approaches for unbiased visual emotion recognition. Code is available at https://github.com/donydchen/causal_emotion