mlsanityData to 7 Oct 2026

ArtEmis: Affective Language for Visual Art

arXiv 2101.07396 ↗First posted 19 Jan 2021cs.CV cs.CLcited by 103

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a novel large-scale dataset and accompanying machine learning models aimed at providing a detailed understanding of the interplay between visual content, its emotional effect, and explanations for the latter in language. In contrast to most existing annotation datasets in computer vision, we focus on the affective experience triggered by visual artworks and ask the annotators to indicate the dominant emotion they feel for a given image and, crucially, to also provide a grounded verbal explanation for their emotion choice. As we demonstrate below, this leads to a rich set of signals for both the objective content and the affective impact of an image, creating associations with abstract concepts (e.g., "freedom" or "love"), or references that go beyond what is directly visible, including visual similes and metaphors, or subjective references to personal experiences. We focus on visual art (e.g., paintings, artistic photographs) as it is a prime example of imagery created to elicit emotional responses from its viewers. Our dataset, termed ArtEmis, contains 439K emotion attributions and explanations from humans, on 81K artworks from WikiArt. Building on this data, we train and demonstrate a series of captioning systems capable of expressing and explaining emotions from visual stimuli. Remarkably, the captions produced by these systems often succeed in reflecting the semantic and abstract content of the image, going well beyond systems trained on existing datasets. The collected dataset and developed methods are available at https://artemisdataset.org.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 33 of its 73 references that are in the map. First posted on arXiv 19 Jan 2021.

Builds on

33 of 73 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Fast R-CNNcited by 2,930
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
21 more
  1. 2017Focal Loss for Dense Object Detectioncited by 749
  2. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  3. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  4. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  5. 2019Connecting Vision and Language with Localized Narrativescited by 225
  6. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  7. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  8. 2019Strategies for Structuring Story Generationcited by 124
  9. 2018Neural Baby Talkcited by 107
  10. 2018How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrievalcited by 73
  11. 2016Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmarkcited by 71
  12. 2017Colors in Context: A Pragmatic Neural Model for Grounded Language Understandingcited by 57
  13. 2019ShapeGlot: Learning Language for Shape Differentiationcited by 54
  14. 2018Learning to Evaluate Image Captioningcited by 53
  15. 2015SentiCap: Generating Image Descriptions with Sentimentscited by 48
  16. 2016Re-evaluating Automatic Metrics for Image Captioningcited by 45
  17. 2017Context-aware Captions from Context-agnostic Supervisioncited by 43
  18. 2017BAM! The Behance Artistic Media Dataset for Recognition Beyond Photographycited by 43
  19. 2017Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networkscited by 14
  20. 2018Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptionscited by 12
  21. 2019Prose for a Paintingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 13; 2023: 20; 2024: 19; 2025: 16; 2026: 28.

103 papers.

Sort:

By area:

  1. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  4. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  5. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
  6. 2021Understanding and Creating Art with AI: Review and Outlookcited by 60
  7. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  8. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  9. 2024AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perceptioncited by 53
  10. 2023Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Methodcited by 48
  11. 2021Language-Driven Image Style Transfercited by 36
  12. 2022It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collectioncited by 36
38 more
  1. 2025ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understandingcited by 36
  2. 2021ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similaritycited by 34
  3. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  4. 2022MetaCLUE: Towards Comprehensive Visual Metaphors Researchcited by 28
  5. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  6. 2024No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languagescited by 19
  7. 2022ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culturecited by 17
  8. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  9. 2025LAPIS: A novel dataset for personalized image aesthetic assessmentcited by 16
  10. 2023ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adaptercited by 14
  11. 2022StyleBabel: Artistic Style Tagging and Captioningcited by 12
  12. 2023Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotioncited by 12
  13. 2024The Call for Socially Aware Language Technologiescited by 12
  14. 2025EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessmentcited by 12
  15. 2024Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understandingcited by 11
  16. 2025Affective Image Editing: Shaping Emotional Factors via Text Descriptionscited by 11
  17. 2021AffectGAN: Affect-Based Generative Art Driven by Semanticscited by 10
  18. 2025EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generationcited by 10
  19. 2022How Would The Viewer Feel? Estimating Wellbeing From Video Scenarioscited by 9
  20. 2023Prompt Evolution for Generative AI: A Classifier-Guided Approachcited by 9
  21. 2023OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?cited by 9
  22. 2024StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Modelscited by 8
  23. 2024Unveiling the Invisible: Captioning Videos with Metaphorscited by 7
  24. 2025ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understandingcited by 7
  25. 2022FastCLIPstyler: Optimisation-free Text-based Image Style Transfer Using Style Representationscited by 6
  26. 2023Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image Methodscited by 6
  27. 2025EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysiscited by 6
  28. 2025Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priorscited by 6
  29. 2023ImageCaptioner2: Image Captioner for Image Captioning Bias Amplification Assessmentcited by 5
  30. 2023Creative Data Generation: A Review Focusing on Text and Poetrycited by 5
  31. 2023Socratis: Are large multimodal models emotionally aware?cited by 5
  32. 2023Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversationscited by 4
  33. 2024Evaluation in Neural Style Transfer: A Reviewcited by 4
  34. 2024Training A Small Emotional Vision Language Model for Visual Art Comprehensioncited by 4
  35. 2024KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graphcited by 4
  36. 2023ViPE: Visualise Pretty-much Everythingcited by 3
  37. 2023Impressions: Understanding Visual Semiotics and Aesthetic Impactcited by 3
  38. 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoningcited by 3

50 most cited of 103 All 103 (JSON)