# Aesthetically Relevant Image Captioning

arXiv: https://arxiv.org/abs/2211.15378 · HTML: https://mlsanity.com/paper/2211.15378/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 25 Nov 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [Affective Image Content Analysis: Two Decades Review and New Perspectives](https://mlsanity.com/paper/2106.16125/) (2021)
- [Aesthetic Attributes Assessment of Images](https://mlsanity.com/paper/1907.04983/) (2019)
- [Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment](https://mlsanity.com/paper/2206.08614/) (2022)
- [Will People Like Your Image? Learning the Aesthetic Space](https://mlsanity.com/paper/1611.05203/) (2016)
- [Aesthetic Image Captioning From Weakly-Labelled Photographs](https://mlsanity.com/paper/1908.11310/) (2019)

## Built on by (most cited first, 6 of 6)

- [AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception](https://mlsanity.com/paper/2401.08276/) (2024)
- [The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographers](https://mlsanity.com/paper/2509.18582/) (2025)
- [Omnidirectional Image Quality Captioning: A Large-scale Database and A New Model](https://mlsanity.com/paper/2502.15271/) (2025)
- [Language-Guided Visual Perception Disentanglement for Image Quality Assessment and Conditional Image Generation](https://mlsanity.com/paper/2503.02206/) (2025)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Aesthetic Image Captioning with Saliency Enhanced MLLMs](https://mlsanity.com/paper/2509.04378/) (2025)

## Abstract (arXiv)

Image aesthetic quality assessment (AQA) aims to assign numerical aesthetic ratings to images whilst image aesthetic captioning (IAC) aims to generate textual descriptions of the aesthetic aspects of images. In this paper, we study image AQA and IAC together and present a new IAC method termed Aesthetically Relevant Image Captioning (ARIC). Based on the observation that most textual comments of an image are about objects and their interactions rather than aspects of aesthetics, we first introduce the concept of Aesthetic Relevance Score (ARS) of a sentence and have developed a model to automatically label a sentence with its ARS. We then use the ARS to design the ARIC model which includes an ARS weighted IAC loss function and an ARS based diverse aesthetic caption selector (DACS). We present extensive experimental results to show the soundness of the ARS concept and the effectiveness of the ARIC model by demonstrating that texts with higher ARS's can predict the aesthetic ratings more accurately and that the new ARIC model can generate more accurate, aesthetically more relevant and more diverse image captions. Furthermore, a large new research database containing 510K images with over 5 million comments and 350K aesthetic scores, and code for implementing ARIC are available at https://github.com/PengZai/ARIC.