# Aesthetic Image Captioning with Saliency Enhanced MLLMs

arXiv: https://arxiv.org/abs/2509.04378 · HTML: https://mlsanity.com/paper/2509.04378/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining](https://mlsanity.com/paper/2303.14302/) (2023)
- [AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception](https://mlsanity.com/paper/2404.09624/) (2024)
- [Effective Aesthetics Prediction with Multi-level Spatially Pooled Features](https://mlsanity.com/paper/1904.01382/) (2019)
- [UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark](https://mlsanity.com/paper/2404.09619/) (2024)
- [Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment](https://mlsanity.com/paper/2004.03015/) (2020)
- [Texture-guided Saliency Distilling for Unsupervised Salient Object Detection](https://mlsanity.com/paper/2207.05921/) (2022)
- [Aesthetic Attributes Assessment of Images](https://mlsanity.com/paper/1907.04983/) (2019)
- [Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment](https://mlsanity.com/paper/2206.08614/) (2022)
- [UniQA: Unified Vision-Language Pre-training for Image Quality and Aesthetic Assessment](https://mlsanity.com/paper/2406.01069/) (2024)
- [Aesthetic Image Captioning From Weakly-Labelled Photographs](https://mlsanity.com/paper/1908.11310/) (2019)
- [Neural Aesthetic Image Reviewer](https://mlsanity.com/paper/1802.10240/) (2018)
- [Co-Salient Object Detection with Co-Representation Purification](https://mlsanity.com/paper/2303.07670/) (2023)
- [Aesthetically Relevant Image Captioning](https://mlsanity.com/paper/2211.15378/) (2022)
- [Image Aesthetics Assessment Using Graph Attention Network](https://mlsanity.com/paper/2206.12869/) (2022)
- [KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph](https://mlsanity.com/paper/2409.10921/) (2024)

## Abstract (arXiv)

Aesthetic Image Captioning (AIC) aims to generate textual descriptions of image aesthetics, becoming a key research direction in the field of computational aesthetics. In recent years, pretrained Multimodal Large Language Models (MLLMs) have advanced rapidly, leading to a significant increase in image aesthetics research that integrates both visual and textual modalities. However, most existing studies on image aesthetics primarily focus on predicting aesthetic ratings and have shown limited application in AIC. Existing AIC works leveraging MLLMs predominantly rely on fine-tuning methods without specifically adapting MLLMs to focus on target aesthetic content. To address this limitation, we propose the Aesthetic Saliency Enhanced Multimodal Large Language Model (ASE-MLLM), an end-to-end framework that explicitly incorporates aesthetic saliency into MLLMs. Within this framework, we introduce the Image Aesthetic Saliency Module (IASM), which efficiently and effectively extracts aesthetic saliency features from images. Additionally, we design IAS-ViT as the image encoder for MLLMs, this module fuses aesthetic saliency features with original image features via a cross-attention mechanism. To the best of our knowledge, ASE-MLLM is the first framework to integrate image aesthetic saliency into MLLMs specifically for AIC tasks. Extensive experiments demonstrated that our approach significantly outperformed traditional methods and generic MLLMs on current mainstream AIC benchmarks, achieving state-of-the-art (SOTA) performance.