mlsanityData to 7 Oct 2026

Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion

arXiv 2307.13463 ↗First posted 25 Jul 2023cs.CV cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The emergence of artificial emotional intelligence technology is revolutionizing the fields of computers and robotics, allowing for a new level of communication and understanding of human behavior that was once thought impossible. While recent advancements in deep learning have transformed the field of computer vision, automated understanding of evoked or expressed emotions in visual media remains in its infancy. This foundering stems from the absence of a universally accepted definition of "emotion", coupled with the inherently subjective nature of emotions and their intricate nuances. In this article, we provide a comprehensive, multidisciplinary overview of the field of emotion analysis in visual media, drawing on insights from psychology, engineering, and the arts. We begin by exploring the psychological foundations of emotion and the computational principles that underpin the understanding of emotions from images and videos. We then review the latest research and systems within the field, accentuating the most promising approaches. We also discuss the current technological challenges and limitations of emotion analysis, underscoring the necessity for continued investigation and innovation. We contend that this represents a "Holy Grail" research problem in computing and delineate pivotal directions for future inquiry. Finally, we examine the ethical ramifications of emotion-understanding technologies and contemplate their potential societal impacts. Overall, this article endeavors to equip readers with a deeper understanding of the domain of emotion analysis in visual media and to inspire further research and development in this captivating and rapidly evolving field.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 104 of its 497 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

104 of 497 references.

References · 104

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
38 more
  1. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  6. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  7. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  8. 2017A Survey on Multi-Task Learningcited by 838
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  10. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  12. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  13. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  14. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  15. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  16. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  17. 2018Deep Facial Expression Recognition: A Surveycited by 269
  18. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  19. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  20. 2017Integral Human Pose Regressioncited by 255
  21. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  22. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  23. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  24. 2018Learning Factorized Multimodal Representationscited by 171
  25. 2020Suppressing Uncertainties for Large-Scale Facial Expression Recognitioncited by 157
  26. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  27. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  28. 2019Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognitioncited by 127
  29. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  30. 2021ArtEmis: Affective Language for Visual Artcited by 103
  31. 2019Context-Aware Emotion Recognition Networkscited by 99
  32. 2021Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognitioncited by 89
  33. 2021TransFER: Learning Relation-aware Facial Expression Representations with Transformerscited by 85
  34. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  35. 2020Context Based Emotion Recognition using EMOTIC Datasetcited by 73
  36. 2016Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmarkcited by 71
  37. 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 69
  38. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 1.

12 papers.

  1. 2025Evaluating Vision-Language Models for Emotion Recognitioncited by 19
  2. 2024Identity-free Artificial Emotional Intelligence via Micro-Gesture Understandingcited by 8
  3. 2025Visual Dominance and Emerging Multimodal Approaches in Distracted Driving Detection: A Review of Machine Learning Techniquescited by 5
  4. 2026Explainable graph attention network for stress recognition (StressGAT) via differential action unitscited by 5
  5. 2025Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approachescited by 3
  6. 2024DuoSpaceNet: Leveraging Both Bird's-Eye-View and Perspective View Representations for 3D Object Detectioncited by 2
  7. 2025Don't Judge Before You CLIP: A Unified Approach for Perceptual Taskscited by 2
  8. 2025PRJ: Perception-Retrieval-Judgement for Generated Imagescited by 2
  9. 2024Incorporating simulated spatial context information improves the effectiveness of contrastive learning modelscited by 1
  10. 2025A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networkscited by 1
  11. 2025Measuring Anxiety Levels with Head Motion Patterns in Severe Depression Populationcited by 1
  12. 2025VAEER: Visual Attention-Inspired Emotion Elicitation Reasoningcited by 1