# Impressions: Understanding Visual Semiotics and Aesthetic Impact

arXiv: https://arxiv.org/abs/2310.17887 · HTML: https://mlsanity.com/paper/2310.17887/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 73 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Reasoning About Pragmatics with Neural Listeners and Speakers](https://mlsanity.com/paper/1604.00562/) (2016)
- [Engaging Image Captioning Via Personality](https://mlsanity.com/paper/1810.10665/) (2018)
- [Natural Language Understanding with the Quora Question Pairs Dataset](https://mlsanity.com/paper/1907.01041/) (2019)
- [Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs](https://mlsanity.com/paper/2010.07526/) (2020)
- [Context-aware Captions from Context-agnostic Supervision](https://mlsanity.com/paper/1701.02870/) (2017)
- [Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias](https://mlsanity.com/paper/1808.02212/) (2018)
- [A Dataset and Baselines for Visual Question Answering on Art](https://mlsanity.com/paper/2008.12520/) (2020)
- [Attend to You: Personalized Image Captioning with Context Sequence Memory Networks](https://mlsanity.com/paper/1704.06485/) (2017)
- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [MetaCLUE: Towards Comprehensive Visual Metaphors Research](https://mlsanity.com/paper/2212.09898/) (2022)
- [Stereotyping and Bias in the Flickr30K Dataset](https://mlsanity.com/paper/1605.06083/) (2016)
- [Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization](https://mlsanity.com/paper/2212.10397/) (2022)
- [Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning](https://mlsanity.com/paper/2108.11912/) (2021)
- [Evaluating Pragmatic Abilities of Image Captioners on A3DS](https://mlsanity.com/paper/2305.12777/) (2023)
- [Pandora: Description of a Painting Database for Art Movement Recognition with Baselines and Perspectives](https://mlsanity.com/paper/1602.08855/) (2016)

## Built on by (most cited first, 3 of 3)

- [AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception](https://mlsanity.com/paper/2401.08276/) (2024)
- [AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception](https://mlsanity.com/paper/2404.09624/) (2024)
- [ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding](https://mlsanity.com/paper/2507.14533/) (2025)

## Abstract (arXiv)

Is aesthetic impact different from beauty? Is visual salience a reflection of its capacity for effective communication? We present Impressions, a novel dataset through which to investigate the semiotics of images, and how specific visual features and design choices can elicit specific emotions, thoughts and beliefs. We posit that the impactfulness of an image extends beyond formal definitions of aesthetics, to its success as a communicative act, where style contributes as much to meaning formation as the subject matter. However, prior image captioning datasets are not designed to empower state-of-the-art architectures to model potential human impressions or interpretations of images. To fill this gap, we design an annotation task heavily inspired by image analysis techniques in the Visual Arts to collect 1,440 image-caption pairs and 4,320 unique annotations exploring impact, pragmatic image description, impressions, and aesthetic design choices. We show that existing multimodal image captioning and conditional generation models struggle to simulate plausible human responses to images. However, this dataset significantly improves their ability to model impressions and aesthetic evaluations of images through fine-tuning and few-shot adaptation.